Meeting Minutes for 19 April 2006

Apr 19 20:57:13 *    Sonar_Gal now calls this meeting to Order
Apr 19 20:57:42 *    daMaestro sneaks in the back door
Apr 19 20:58:11 *    EvilBob is still looking for the "this meeting" menu
Apr 19 20:58:18 Sonar_Gal    ok
Apr 19 20:58:43 EvilBob    Sonar_Gal: May I have the floor please?
Apr 19 20:59:06 *    Sonar_Gal now give the floor to Bob
Apr 19 20:59:31 EvilBob    daMaestro: the other day you expressed some needs/concerns for the search project, I was not online at the time can you give an update
Apr 19 21:00:05 daMaestro    well i was thinking about dropping one of our WAN links... this has passed
Apr 19 21:00:53 daMaestro    so the inet connectivity is fine
Apr 19 21:01:23 EvilBob    SO all is good or should we still look at moving the hosting of the project?
Apr 19 21:02:23 EvilBob    SO all is good or should we still look at moving the hosting of the project?
Apr 19 21:02:33 daMaestro    for the current time.. it is ok. maybe even in a better place then before. after I had a meeting.. we might even be getting a 100mbit link.. currently we have a 5mbit
Apr 19 21:02:49 daMaestro    however, planning to move the search system is not a bad idea
Apr 19 21:02:49 EvilBob    OK
Apr 19 21:03:12 daMaestro    if unity has the servers.. it could happen
Apr 19 21:03:13 EvilBob    well I do not want to have an issue for the long term
Apr 19 21:03:50 daMaestro    when I look to the long term.. it seems we should plan to have it on a better link
Apr 19 21:04:07 daMaestro    i really hope it becomes _the_ fedora search system
Apr 19 21:04:07 EvilBob    OK so moving would be a good thing
Apr 19 21:04:16 daMaestro    EvilBob, IMO.. yes
Apr 19 21:04:40 daMaestro    EvilBob, though I had a great meeting today.. and we (the company I work with) might be expanding
Apr 19 21:04:43 *    Sonar_Guy finished shortnaming FS.o and FM.o
Apr 19 21:05:10 daMaestro    thus new servers and a 100mbit line to the inet at the main DC
Apr 19 21:05:53 EvilBob    OK well if we are going to do this I will need to dedicate some additional hardware and at least two IPs
Apr 19 21:06:00 daMaestro    EvilBob, the cluster is planned.. on my own personal budget, so I am saving for that, however I have no idea when I will reach my goal
Apr 19 21:06:44 EvilBob    right
Apr 19 21:07:05 EvilBob    For now I think we should be able to get by with a couple of Virtual machines
Apr 19 21:07:23 EvilBob    the only thing we need is some ram and the ips for them
Apr 19 21:07:24 daMaestro    EvilBob, at this time I can not say one way or another. it is really the choice of the group.. I currently have the hardware to support it and the resources
Apr 19 21:08:09 EvilBob    Sonar_Gal: are you guys still willing to sponsor this expansion?
Apr 19 21:08:17 Sonar_Guy    EvilBob: Yes
Apr 19 21:08:27 Southern_Gentlem    what do you need
Apr 19 21:08:32 EvilBob    Sonar_Guy: OK I will call right now and get the ram price
Apr 19 21:08:41 Sonar_Gal    EvilBob, not a problem
Apr 19 21:09:07 daMaestro    EvilBob, I guess I will go on the record and say it would be _better_ to have unity have its own servers for the search system.. I will still manage it and continue to improve it as I work on my search engine
Apr 19 21:09:18 daMaestro    I have massive plans.. but they are moving slowly
Apr 19 21:09:40 EvilBob    daMaestro: that is in our goals
Apr 19 21:10:42 EvilBob    daMaestro: having you take care of the search systems is a must at this time, however I would like to have you teach another how to maintain it for in a worst case situation
Apr 19 21:11:11 Sonar_Gal    +1
Apr 19 21:11:30 Sonar_Guy    +1
Apr 19 21:11:32 EvilBob    We are looking at a $400 outlay for the ram and IP addresses (for 1 year)
Apr 19 21:11:53 Sonar_Gal    ok
Apr 19 21:12:06 daMaestro    EvilBob, the current plan on my end is to get some sort of massive NAS and/or SAN .. using fibre channel or SATAII with 2TB of storage .. and a cluster of amd64 machines with 4GB ram each (3 machines planned at the moment)
Apr 19 21:12:18 EvilBob    the Ips are $5 each per month I say we get 2 for the search system and a 3rd for growth
Apr 19 21:12:26 Sonar_Guy    np
Apr 19 21:12:50 Sonar_Guy    EvilBob: you really need to get a paypal account setup for transfering funds.  hehe
Apr 19 21:13:11 daMaestro    EvilBob, and stage 2 is moving to a 100mbit link.. this will happen (if it does) in 3 months
Apr 19 21:13:47 EvilBob    My suppier that is holding the 2 Gigs of ram for us did not answer
Apr 19 21:14:58 EvilBob    Sonar_Gal: I will get you the exact amounts in the morning
Apr 19 21:15:06 Sonar_Gal    ok not a problem
Apr 19 21:15:33 EvilBob    On Monday I will put the new ram in the server and start building the VMs
Apr 19 21:15:41 daMaestro    EvilBob, the indexing system could run on one server.. and an existing IP.. though this design is not optimal
Apr 19 21:16:06 EvilBob    daMaestro: I think we break it off in to a VM so it is portable
Apr 19 21:16:14 daMaestro    +1
Apr 19 21:16:38 Sonar_Gal    +1
Apr 19 21:16:49 Sonar_Guy    +1
Apr 19 21:17:01 EvilBob    If other hosting comes along that will make life easier
Apr 19 21:17:33 EvilBob    daMaestro: please email me the needed specs "drivespace" and so on for the two machines
Apr 19 21:18:04 daMaestro    EvilBob, give me a sec to check what it is using right now.. the compression is just awesome
Apr 19 21:18:27 EvilBob    daMaestro: we have abotu 200 Gigs available now
Apr 19 21:18:52 daMaestro    /dev/mapper/raid-storage 891G  2.3G  844G   1% /storage
Apr 19 21:19:02 daMaestro    lol.. that is a full index of about a million urls
Apr 19 21:19:12 EvilBob    so 20Gig will be fine
Apr 19 21:19:13 daMaestro    half have full document copies
Apr 19 21:19:30 EvilBob    is that right?
Apr 19 21:19:37 daMaestro    i think more would be better
Apr 19 21:20:01 EvilBob    just tell me what we need and how you want it
Apr 19 21:20:05 Questor    1,000,000 url's is using only 2.3 gigs?
Apr 19 21:20:22 EvilBob    Questor: text is small
Apr 19 21:20:22 daMaestro    50G would be about right.. though I have had issues with throughput
Apr 19 21:20:30 daMaestro    Questor, compressed text is even smaller
Apr 19 21:21:00 daMaestro    EvilBob, making sure we have great read/write speed is the most important
Apr 19 21:21:07 EvilBob    daMaestro: send e an email withhow you would like the partitions laid out
Apr 19 21:21:10 Questor    How many URL's do you estimate eventually indexing?
Apr 19 21:21:29 EvilBob    daMaestro: should not be an issue, we are running on SATA II drives
Apr 19 21:21:34 daMaestro    Questor, well the filtered fedora engine has about 200K
Apr 19 21:21:40 daMaestro    800K are for another project
Apr 19 21:22:06 daMaestro    EvilBob, how hard would it be to add HDD space? 20G will get us up and running and will have room for expansion
Apr 19 21:22:15 daMaestro    EvilBob, I might be looking ahead a little too far
Apr 19 21:23:16 EvilBob    daMaestro: should not be veryhard, I will set up a seperage virtual drive for the data
Apr 19 21:23:16 EvilBob    10Gigs for / XGigs for /storage
Apr 19 21:23:24 daMaestro    EvilBob, ok.. i will think it over and send an email.. however all I need for the search system is a FC5 install (partition however) and a 20G /some/path partition
Apr 19 21:23:41 daMaestro    sounds like a plan
Apr 19 21:23:50 daMaestro    go with 20G to start...
Apr 19 21:23:55 EvilBob    Done
Apr 19 21:24:24 EvilBob    1.5 Gigs Ram for the indexer 512MiB for the frontend?
Apr 19 21:24:49 daMaestro    actaully the cacheD searchD server is what needs the RAM
Apr 19 21:24:56 daMaestro    the indexer just needs a fast clock
Apr 19 21:24:57 EvilBob    Oh ok
Apr 19 21:25:20 Southern_Gentlem    define fast clock
Apr 19 21:25:23 daMaestro    I think it might be a good plan to have the frontend server also be the indexer.. if we only go with 2 VMs
Apr 19 21:25:31 daMaestro    Southern_Gentlem, as fast as we can go
Apr 19 21:26:08 EvilBob    so a Gig for the frontend and 512MiB for the indexer
Apr 19 21:26:08 EvilBob    daMaestro: OK lay it out figure we have 1.5Gigs ram to work with
Apr 19 21:26:12 EvilBob    using 2 VMs
Apr 19 21:26:42 daMaestro    hrmm well the current system uses 3G
Apr 19 21:27:28 daMaestro    I would think a Gig for the searchD/storeD/cacheD server and 512MiB for the frontend/indexer would be the best option
Apr 19 21:27:40 daMaestro    am I going to be able to tune these servers?
Apr 19 21:27:49 daMaestro    I mean pretty much everything will be shut off
Apr 19 21:28:01 Sonar_Guy    Then if we see a need for more memory we can go from there.
Apr 19 21:28:02 EvilBob    max I can go is 4gigs on the host
Apr 19 21:28:02 EvilBob    I am planning on having that in there
Apr 19 21:28:06 EvilBob    daMaestro: yup
Apr 19 21:28:27 EvilBob    If I need to move other VMs off that box I can
Apr 19 21:28:57 daMaestro    EvilBob, i would not expect so.. how fast is the chip?
Apr 19 21:29:18 EvilBob    daMaestro: 2.8Ghz HT P4e
Apr 19 21:29:36 daMaestro    ok.. well there is only one way to find out.. so lets go for it
Apr 19 21:29:51 EvilBob    yup
Apr 19 21:30:10 Sonar_Guy    Then we tweak from there.
Apr 19 21:30:23 EvilBob    Sonar_Gal: the ram is $94 ea, the IPs are $180 for the year
Apr 19 21:30:35 daMaestro    I will work on clustering it on my own time.. when I manage to finish my inventory and clean out what I am not using
Apr 19 21:30:58 Sonar_Guy    EvilBob: so 400 total should cover it?
Apr 19 21:31:06 EvilBob    Sonar_Guy: yup
Apr 19 21:31:08 daMaestro    EvilBob, what connectivity do we have there?
Apr 19 21:31:29 EvilBob    daMaestro: 3 T1 right now with minimal inbound
Apr 19 21:31:46 EvilBob    daMaestro: most everything is outgoing naturally
Apr 19 21:32:02 *    rjune has quit (Remote closed the connection)
Apr 19 21:32:23 daMaestro    EvilBob, well.. if we come up with a creative solution.. we are able to have other machines off site do indexing also.. and just submit data to the cache server
Apr 19 21:32:47 daMaestro    so I think this a good plan to get us started
Apr 19 21:32:47 EvilBob    daMaestro: that is an option with the servers we have here inhouse
Apr 19 21:33:09 EvilBob    daMaestro: I hope to get a cluster here runnign in the next 60 days
Apr 19 21:33:23 daMaestro    EvilBob, that is about my target time too
Apr 19 21:33:39 daMaestro    EvilBob, i could have a sweet one going.. but the power costs are just too much
Apr 19 21:33:43 EvilBob    daMaestro: looking at 24 nodes here
Apr 19 21:33:48 daMaestro    EvilBob, so I need to update my hardware.
Apr 19 21:34:02 daMaestro    EvilBob, sweet.. sounds like this is going to be fun
Apr 19 21:34:55 daMaestro    i have a few investors to the search engine I am working on.. so. well it will be fun
Apr 19 21:37:11 EvilBob    Ok anything else?
Apr 19 21:37:17 Questor    !
Apr 19 21:37:21 Sonar_Guy    Hell, I would just like one of our cabinets for the new sonar system 13 2u 3.06 dual hts with 1 raid array in a 3u 3.06 dual ht.  That would make a helluva cluster!!
Apr 19 21:37:50 Sonar_Gal    Questor, go ahead it's an openfloor meeting
Apr 19 21:37:56 Questor    oh ok
Apr 19 21:38:18 Questor    daMaestro, just wanted to say, I have something to mention to you when the meeting is over.
Apr 19 21:38:20 daMaestro    I need to find a better storage solution.. all my raid is ide.. so boo
Apr 19 21:38:31 daMaestro    Questor, ok sounds like a plan
Apr 19 21:38:55 daMaestro    EvilBob, how soon are we going to go live?
Apr 19 21:39:12 EvilBob    daMaestro: I will have the machines set up on Monday
Apr 19 21:39:22 daMaestro    EvilBob, I like the current settings.. I will continue to maintain a devel system.. so I can break it when I feel like it
Apr 19 21:39:23 Questor    Sonar_Guy, can you translate this to English:  "13 2u 3.06 dual hts with 1 raid array in a 3u 3.06 dual ht." ?
Apr 19 21:39:35 daMaestro    Questor, s.e.x.y.
Apr 19 21:40:21 Questor    lol
Apr 19 21:40:57 daMaestro    basically 13 3GHz hyper threaded (HT) P4s (i assume) connected to one storage array on a dual 3GHz HT machine
Apr 19 21:41:19 Sonar_Guy    13 2u SMPs with dual intel 3.06 GHZ HT processors with a 3u SMP with the same specs as the 2u units, all smps have 4 gb ram for each processor.  Raid array is 4 250gb hot-swap scsi drives.
Apr 19 21:42:04 daMaestro    Questor, like i said "sexy"
Apr 19 21:42:33 Sonar_Guy    running FC2 btw.
Apr 19 21:42:40 Questor    Ah.
Apr 19 21:42:50 Questor    "FC2" I understand.  :)
Apr 19 21:43:21 Sonar_Guy    They were gonna do RH 6.3 but I knocked some sense into them.
Apr 19 21:43:24 Sonar_Gal    Questor, why do you think it's so easy for him to trouble shoot in legacy
Apr 19 21:43:42 Questor    What's a 2u and a 3u?
Apr 19 21:43:58 daMaestro    a measurement of case size.. normally rack mounted
Apr 19 21:44:04 EvilBob    Questor: height measurement
Apr 19 21:44:21 EvilBob    Questor: 1U = about 1.5"
Apr 19 21:44:59 daMaestro    EvilBob, what sort of VPN support do we have?
Apr 19 21:45:01 Sonar_Guy    Yep gives us room for 3 Cisco 7550 gigabit routers in the top of the cabinet.
Apr 19 21:45:32 EvilBob    daMaestro: what do you want? I do everythign over SSH
Apr 19 21:46:10 Sonar_Guy    daMaestro: what you would really like is our I/O server 20 64bit PCI slots with quad 3.06 ghz HT processors.
Apr 19 21:46:55 Sonar_Guy    Custom of course, Next time I get up to the lap I will get some pics for you!!
Apr 19 21:47:05 Questor    So I get 13 3" Symmetric MultiProcessing 3.06 gigahertz hyper-threaded Intel Pentium 4 CUPS, with one more 4.5" Symmetric Multiprocessing computer 3.06 GHz dual-processor Hyperthreaded Pentium 4 CPU, with each unit havint 4 gigabytes of RAM (for a total of 56 gigabytes of memory for thwhoel things).  Add to that four 250 gigabyte hot-swap SCSI hard drives on the 4.5" unit...
Apr 19 21:47:14 Questor    CUPS -> CPUS
Apr 19 21:47:26 daMaestro    EvilBob, for when.. and if.. we setup off site indexers.. the system will need to be able to securely work with the cacheD and storeD.. and not have to change _anything_.. so some sort of transport layer security is good
Apr 19 21:47:41 Sonar_Guy    Questor: YUP, SEXY!!
Apr 19 21:48:02 EvilBob    daMaestro: betweent he machiens should not be an issue
Apr 19 21:49:11 Questor    daMaestro, how soon do you estimate needing off-site indexers?
Apr 19 21:49:12 daMaestro    EvilBob, right.. because it will all be local.. suppose it would be behind some sort of firewall.. but I would like to keep the option open of having off site indexers
Apr 19 21:49:21 Sonar_Guy    Questor: actually 112gb of ram in the rack.
Apr 19 21:49:30 Sonar_Guy    4gb per cpu.
Apr 19 21:50:12 EvilBob    daMaestro: behind the firewall yes
Apr 19 21:50:25 daMaestro    Questor, if we do things right.. I am going to try to index the world for fedora content.. not expect it to be on certain sites.
Apr 19 21:51:19 Questor    daMaestro: Mr. FedoraGoogle?
Apr 19 21:51:41 daMaestro    Questor, the goal of the search system (at least one of my goals) is to create new.. updated content dynamically using what the search system knows about
Apr 19 21:51:56 Questor    :)
Apr 19 21:52:10 daMaestro    and then add in a peer review network.. and bam..
Apr 19 21:52:22 daMaestro    but lets get stage one done
Apr 19 21:53:04 Questor    I guess that scraps using del.icio.us then for getting URL's...
Apr 19 21:53:23 daMaestro    Questor, i actaully think that is a good idea.. no reason to not accept what is..
Apr 19 21:53:37 daMaestro    Questor, would you bring that idea to the group
Apr 19 21:53:44 Questor    Now?
Apr 19 21:53:55 daMaestro    Questor, yeah.. we are in a meeting.. this would be a good time
Apr 19 21:53:59 Questor    Okay.
Apr 19 21:54:52 EvilBob    I think we are done witht he search issues
Apr 19 21:54:56 Questor    daMaestro & I were talking the other day, and I proposed that a good source of Fedora-related and sort-of peer-reviewed  URL's for Fedora Content could come from  ummm...  http://del.icio.us/
Apr 19 21:55:05 EvilBob    any new ideas
Apr 19 21:55:14 Questor    um
Apr 19 21:55:16 Questor    ok
Apr 19 21:55:18 Questor    I'll shut up.
Apr 19 21:55:36 EvilBob    Questor: Go for it
Apr 19 21:56:16 Questor    Do you all know what del.icio.us is/does?
Apr 19 21:56:21 EvilBob    Nope
Apr 19 21:56:23 Southern_Gentlem               no
Apr 19 21:56:24 Sonar_Guy    Nope
Apr 19 21:57:08 daMaestro    i have another item after this.. but I do think this is a good idea and I think I have a solution for getting the urls into the search system.. but go on Questor
Apr 19 21:57:15 Questor    Basically, del.icio.us is a community set of bookmarks, where people download their bookmark set from their web-browsers
Apr 19 21:57:26 Questor    and then tag each of their bookmarks with relevant tags.
Apr 19 21:57:45 Questor    So to see all of the bookmarks that people have put in and tagged "Fedora", for example,
Apr 19 21:58:29 Questor    one merely need to to this URL:   http://del.icio.us/tag/fedora
Apr 19 21:59:29 Questor    This webpage orders them by most recent to oldest... but you may note that they are ranked as well.
Apr 19 22:00:08 Questor    The page has an RSS feed so such data can be slurped up to help inform any search engines or whatever...
Apr 19 22:01:05 Questor    Haven't looked into it really deeply though..
Apr 19 22:02:17 Sonar_Gal    Questor, you will have to give us time to really check out the site then we can dicuss this at another meeting
Apr 19 22:02:23 Questor    Popularly-bookmarked Fedora pages can be gotten here:  http://del.icio.us/popular/fedora
Apr 19 22:02:25 Questor    You bet.
Apr 19 22:02:44 *    EvilBob is slightly distracted working on the new machine
Apr 19 22:02:46 Questor    Just thought you might find it interesting, as another source of info for the unity sites.
Apr 19 22:02:56 EvilBob    yup
Apr 19 22:03:02 Questor    Plugged it in yet, EvilBob?
Apr 19 22:03:08 EvilBob    sounds like a potential resource
Apr 19 22:03:30 EvilBob    yeah CD1 is in, need ot partition some drives
Apr 19 22:03:57 daMaestro    ok.. well we need to find a way to submit revisions to documents
Apr 19 22:04:32 Questor    Doesn't plone track revisions?
Apr 19 22:04:35 daMaestro    i mean copy and paste to a new doc works.. but there have been a few times when there could be info added by random people.. that could be usefull.. not sure if plone has support
Apr 19 22:05:04 daMaestro    yes.. but for non-admin users to be able to submit changes
Apr 19 22:05:18 daMaestro    proposed changes
Apr 19 22:05:21 Questor    Ah, wiki-like?
Apr 19 22:05:25 ignacio    Any CMS without versioning support is just a glorified gedit.
Apr 19 22:05:42 daMaestro    yes.. i guess wiki like editing
Apr 19 22:06:28 daMaestro    i have not looked.. but would this feature be welcomed?
Apr 19 22:06:40 EvilBob    daMaestro: it is needed
Apr 19 22:07:30 Sonar_Gal    ok Any other topics for this meeting
Apr 19 22:07:56 Questor    font size?
Apr 19 22:07:59 EvilBob    Nope
Apr 19 22:08:05 EvilBob    Close it up
Apr 19 22:08:11 Sonar_Gal    lol
Apr 19 22:08:24 *    Sonar_Gal now calls this meeting to a close.