Greetings,
Unfortunately, the John Edwards Keynote at Gnomedex was overly political. As Chris Pirillo put it, once you open up the session to the audience, the audience drives it to where they want it to go.
For the most part it was political questions, comments on the democrat party, and grandstanding by audience members.
Back in the day, I used to have access to the Congressional Record computer system, an VAX/VMS system. One of the interesting things it had was variations of bills as they went through various stages. One thing I learned back then was that all bills are ‘patches’ to existing government code. So the obvious question becomes how that is visualized, written, edited, and collaborated on by the Senate right now. Further, is there any way to expose that inner workings, and the steps involved publicly?
Specifically taking the wiki approach, adding in added/deleted/changed hilighting, along with version control and version identification. It would be IMMENSELY valuable to see which Senator made individual changes in the bills, for instance. The other related question, is how Senators collaborate on writing a bill. For instance, since it’s all deltas to the government code, you have to keep track of what those changes are, and allow multiple disconnected Senators to send data back and forth in order to build the full text of the bill. This seems like a great place for wiki technology to be useful to improve the processes of government.
Anyhow, that was my ‘technical’ question, as opposed to the majority of the political questions/discussions.
We had a good chance to talk real technology with John Edwards, but it devolved more or less to a political mess, and I didn’t even get to ask my question.
Meh.
– Morgan Schweers, CyberFOX!
cynicalGreetings,
So I’m very, very much looking forward to Gnomedex, coming up at the end of this month. I went last year, and it was really cool. I went to the Seattle MindCamp, which was intense, a lot of fun, and tiring. I met good people at both, although the MindCamp folks were more…relaxed, I think. The knowledge sharing was much more peer-to-peer at MindCamp, and I really reveled in it.
Gnomedex is looking to be even more interesting than the last one, with the addition of Senator John Edwards as keynote speaker. I don’t expect much from him in particular, but the added attention it’ll give the conference should be interesting.
More than anything else, I’m looking for the little moments, like last year when I watched the JotLive Wiki updates stuff happening as the guys debugged it while sitting in the back row of the conference. In retrospect, it presaged the Google Spreadsheets interactive/shared updating. The big presentations on stage generally came to nothing. The little moments, and the side discussions are where the action was, and I expect that to be true this time as well.
Plus I get to show off my new house to some friends who are coming into town for the conference!
I need to get my act together, get some software updates out there, and psych myself up for the conference. I’m not a big name guy, so my contributions tend towards having useful pieces of technology to show off and play with.
On a more depressing note, my mother asked for money again, which coincides with the start of my paying mortgage payments, NOT a good time for this. However, I think I may re-release JBidwatcher with the affiliate program enabled again, and funnel the money from it to help her out. I could simply sell the program, but without being able to use the eBay API I can’t promise users it’ll be consistently functional, and that seems like a sucky thing to do to paying customers. At the same time, the program drives revenue to eBay, so it should be of value to them.
This is yet another place that I can get some help at Gnomedex, not so much directly for my mother, but to get an idea of ways that people have monetized their passions outside of work, in order to get an extra income when necessary.
My goal at this years Gnomedex is to keep my ears open; to hear the rumblings of what will make next years tech news, when writ larger. I need to keep in mind the small conversations, and imagine them happening in the halls at Google, or Microsoft, and imagine what they’d do with them…
So many people probably think that Gnomedex will be all about Podcasting, and VLogging. I think that’s just the cover story. The medium(s) are not the message. I think it’s about new ideas, new directions, smart people, and listening carefully to what people are showing off.
Yes, a dozen video hosting sites are making history, and providing all sorts of interesting value, as well as amusing video. Yes, podcasts are so mainstream they’re available on iTunes Music Store, CBS has turned over its incredibly valuable properties into $1.99 iPod Videos, and Suicide Girls post iPod Video-compatible introductions. Perhaps the abundance of new video sources will push out the television, and eventually replace it. This actually interests me less, because it’s a Big Media world.
I’m more interested in what people are saying, rather than how they are saying it. I don’t so much care about the media they use, I care about the ideas they’re getting across.
I probably won’t vlog, I’m not photogenic enough. I don’t really podcast, for the same reason, nobody wants to hear my ideas interspersed with ‘umm’, and such. If anyone is interested at all, they want the thoughts filtered, cleaned up, parsed, and posted, so they can be searched, summarized, stored, and (most often) skipped.
If there’s any serious value out there, in interesting but HARD technology, it’d be in automatically extracting the text of a voice discussion, describing the scene in a video, etc., so it’s searchable using text search engines. This is a Hard Problem, but it would suddenly make the world of VLogs, Podcasts, and even TV and movies, all searchable, indexable, and accessible to everyone.
There’s a world-changing technology, if ever there was one.
See? I’m already getting my mind into the mode for the Gnomedex…
– Morgan Schweers, CyberFOX!
Greetings,
Well, after a few delays and such I’ve managed to make it here. There’s a HUGE traffic jam in front of the ’sign up’ sheets, for people to sign up and people looking over the list of things ‘to do’ over the next day. It’s not worth diving in, just yet, as eventually it’ll clear up.
Parking was sub-optimal, I ended up parking an uncomfortable block away and dragging my super-heavy bag into the center. (Perhaps carrying so much soda and munchies as well as my laptop and PSP wasn’t such a good idea; they appear to have munchies provided…)
Anyhow, it’s a definitely interesting crowd of people, lots of fun conversations floating past. We’ll see if it calms down a bit into a more active conversation, or remains the chaos it appears to be right now…
– Morgan Schweers, CyberFOX!
Greetings,
Wooof… Well, MindCamp2.0 is a minute away from opening up, theoretically, and I’m about 30 minutes away from being there.
I’m really hoping this lets me immerse myself in a crowd of very smart people thinking about cool stuff, so that I can kick-start my own brain cells into working a bit better on my personal projects.
If I talk about anything, it’ll probably be my current attempt at working with NLP and specifically extracting information about whether two blogs talk about similar things. Effectively beating at the ‘information overload’ problem.
Beyond that, we’ll see what happens. I’m hoping I can stay awake for a lot of it. I’ve got my toys (PSP, MacBookPro, Camera), and my pillow.
I’m nervous, but excited. Time to get going!
– Morgan Schweers, CyberFOX!
excitedGreetings,
“…and if you don’t stop and look around you might miss it.”
– Ferris Bueller, Ferris Bueller’s Day Off
This is a purely personal blog entry, so if you’re looking for technical info (especially on eBay’s latest breakage), that’ll have to wait until my next entry.
However, I want to thank all my JBidwatcher users who emailed me to congratulate me on my wedding. I’m so amazingly glad to have produced and supported something such that people wish me well in the other areas of my life. The well-wishes were overwhelming, and made me incredibly grateful for the folks who use JBidwatcher. I am a lucky man, in so many different ways. Thank you.
I’m married now, as of January 28th, 2006. The wedding itself was excellent. We should be getting the official pictures soon, but the day had started out (like many days this January in Seattle) pouring rain. Around noon that day, though, the skies started to clear up. By the time our ceremony rolled around (sunset), the skies were dramatic, gorgeous, and we had a perfect view of them over the water from our ceremony site. The reception was excellent, delicious food, and having my friends there with me kept the stress level just low enough that I could really relax and enjoy the night. For our honeymoon, we went to DisneyWorld (stayed at the Wilderness Lodge) for 5 days, then on the Disney Cruise for 6 days. We were incredibly happy with our decision to put our honeymoon in the hands of Disney, the whole experience was magical, and made it incredibly easy for both of us to focus on each other, and relax from the stress of setting up a wedding entirely ourselves.
On return, I’ve had to re-focus on work a lot, as taking a bit more than 3 weeks off to get married and have a good honeymoon ended up putting my projects in a little bit of danger. However, recently we’ve started seriously looking for a house, and cleaning up our credit in preparation for taking out a mortgage. What with everything that’s been going on, and the everyday things I have to do, it’s been a while since I’ve been able to sit back, take stock, and think about where I’ve been. It’s been an incredible ride, but you have to pause to fix the moments in your memory, or they drift away with the morning fog, and that would be a terrible loss.
I have some pictures I took on our honeymoon, and I’ll put those up eventually, but they are scenes, flashes, images, not the whole, the experience. They are best used to trigger memories, not as memories themselves. They are, in fact, mostly meaningless without context. A reflection of my photography skill, perhaps, but me being part of those memories, I can’t imagine how I would be both outside, looking in, and also have actually enjoyed those moments at the same time.
I’ll be writing more about my technical projects (reflected in the blog), and JBidwatcher as I get time, as those are things I want to focus on for the immediate future.
The mortgage and house-buying process may get some coverage also, as it’s one of the most complex things I’ve had to do in a long time, and uncomfortable for someone used to purely living within their means up until now.
I’ve forgotten how nice it is to try to focus my thoughts, to put together a blog entry. I’m not a prolific blogger like Scoble, or any of the poli-blogs, so I find my writing tries (emphasis on the ‘tries’) to convey more, to make up for the lack of regular updating.
In any case, I am now happily, ecstatically, wonderfully married. That’s enough for now.
Everything else will come in time.
– Morgan Schweers, CyberFOX!
happyGreetings,
Robert Scoble takes Mark Lucovsky to task over seeing passion in Google workers sticking around until all hours of the night.
This is a hard thing to explain if you haven’t been there. I’ve been there twice, once with McAfee Associates, in full-bore, turbo-charged engineer mode, fighting against the world-wide virus writing epidemic in the very early days of McAfee Associates (from 7 people to more than 120 people). Fewer people would recognize that world as would recognize the world of a modern web site developer, so I’ll focus on the second, PayPal.
I remember working at PayPal during their heyday just before and just after going public, when we were fighting the ‘good fight’ against eBay, and I’d work all hours of the day and night not because I had to, but because it was deeply, personally important to me that we win, and because the work was so deeply enthralling that I lost track of time entirely. That everything be right, and that we be first to market with features, that our code be spectacular, that we be innovative and brilliant and FAST was our world. And until we were bought by eBay finally (demonstrating, imo, that we had the better service), no matter the hour, I never was alone at the office.
When you’re part of a brilliant team, trying to honest to god change the world, it’s not about deadlines. It’s about a form of love. It can be thoroughly, caustically destructive to everything else in your life, but it’s an experience I would be a lesser person if I had missed.
If you’re clocking hours at work, and the passion of what you’re doing isn’t keeping you rooted to your chair at all hours, loving the pure joy of creating, fighting the good fight, and trying to change the world, it doesn’t mean it’s not a good job. It’s just not THAT experience.
I promise, there’s far more call for people who work regular hours, meeting normal deadlines, doing solid, good work, than for those of us who burn so very, very brightly, but for so short a time.
If you’re really in passionate love with the work you’re doing, it’s not about working to 3am to meet a deadline. It’s about finally reaching a temporary point of closure for the days work, and raising up your head to suddenly discover it’s 3am.
And if you really, truly believe in your company, and you believe in your project, and have a fire to ‘win’ in some way (usually against a more powerful competitor) it’s not about accepting an imposed deadline that makes you work hard. It’s about DEMANDING a deadline that makes you work hard, but that you know you can meet. Because you know it’s important, and that every second counts, and you CARE about the company being not just first, but first with a brilliant, innovative, wonderful experience.
After it’s all over, it’s draining. It’s exhausting. It’s mind-numbing. You feel…dead, somehow, once the work is over, and you’ve been brilliant for so long, that you feel like your brain cells have used up all their energy. You go home, don’t show up to work for a week, recharge, find out if you still have any RL friends, do something physical (skydiving, rock climbing, hiking, etc.) to get in touch with your body again. You come back to work eventually, and you work with others to clean up any loose ends, and slowly you get back the energy from your co-workers, and the ambience in the office, and eventually you’re back on track to start another feature that’ll knock the socks off your competitors.
If you’re in a company where you are the dominant force, you don’t work like that. You don’t need to, the hunger isn’t there.
PayPal lost that hunger when eBay bought us. There wasn’t anything to fight for, anymore. We’d won, in a way, and lost in a way. I even asked it, when eBay management had a big meeting with everybody to tell us about their vision for us. I don’t remember if it was the meeting Meg Whitman was at, or not, but I asked something like, “We’ve been fighting eBay for all this time, and now we don’t have to. What will replace that, to keep the drive going?” The answer was a mealy-mouthed mess of future strategy and becoming the dominant payment platform. It wasn’t a battle anymore, we’d become the big company.
I left not terribly long after that, for health reasons. (Remember what I said about caustically destructive?
) But also because I didn’t feel the passion in the hallways anymore.
The woman who I will marry in 36 days stood by me through it despite almost never seeing me, my friends teased that they’d forgotten my name, I ended up needing major surgery for a condition I let go too long… But I was part of one of those winning teams, fighting against terrible odds, doing brilliant work, burning so very, very brightly, and changing the world one line of code at a time.
I think Mark understands that, as Google has Microsoft with an unlimited war-chest bearing down on them. From what I’ve read, I don’t think Scoble completely does get it. He gets that passion is important (Channel 9 certainly shows that), but the fight against overwhelming odds that drives it to fevered peaks, that brings it to a different level…that’s what’s missing.
But it’s okay. Even I’m working a day job these days. I still have the intense passion to program on my own projects, doing it until 4am regularly, but I’m in a larger cycle of recharge, get in touch with life, etc., before maybe doing it again if I find the right company. Or maybe not. I’ll be a married man shortly, settling down in theory. Maybe I can’t fight those fights anymore. Maybe I should work at Microsoft.
Just kidding…
– Morgan Schweers, CyberFOX!
contemplativeGreetings,
I’m sitting in a castle which is sitting on a hill, and before me is spread a beautiful green city, tall spires, rolling terrain, glittering buildings, and a harbor to the world.
Between me and the city flows a curtain of white, falling from the sky, making this already fairy-tale city even more so.
I’m sure on the ground, the people of this land are cursing, slipping, and hunching their shoulders against the cold, seeing the trees that are in their way, and not the glory of the forest of beauty surrounding them.
But for this moment, in my red stone tower on a hill, gazing out a window, all is peaceful, snowy, and beautiful.
Seattle in the snow.
– Morgan Schweers, CyberFOX!
calmGreetings,
So, like many geeks, I’ve experimented with the blogging, content-management, etc., thing for a while. I got onto LiveJournal when I was relatively new, because most of the people I knew were on it.
Then, one day, I decided I wanted to set up a version of LiveJournal for my then work company, PayPal (now a division of eBay), so that we could set up some simple work-blogging, so I could easily post up the status of what I’d worked on each day. I chose blogging software because it’s 1:many, where the many are self-selected. I chose LiveJournal because it was the only one I knew of (at that time) which was open source, and supported any number of users.
For a test, I set it up on my home computer, under a virtual host name that I had picked up a while ago. (I collect useful and interesting domain names, not to resell or anything, but because I have an idea to put on them.) The domain name has a meaning to (a specific subset)^3 of science fiction fans, but I didn’t think anything of it. I set up the basic install, configured it, tested it, determined what I would have to do to make it useful at work, and promptly forgot about it. I set the one up at work, nobody used it, management wouldn’t get behind it and encourage the idea, so it died.
That was roughly two years ago.
Today I’m setting up another virtual domain on the same host, this one for my wedding information. To my immense surprise, checking the logs, I find that people are going to that site. It turns out I left the registration system open (as it was how I intended the work system to be), and some people randomly typed the domain name AND were interested enough to create themselves an account. And validate their emails. And post. And bring other people on. And create their own little social network on this…test site.
I haven’t touched it in two years (didn’t even remember it existed!), it has over 250 users, and continues to be a functional community site, for a very small, accidentally selected community.
Some people say that the barrier to creating a successful service on the internet is high; I disagree. I’ve found that it’s possible to create a community site by accident!
In all seriousness, though, it is a testament to the desire people have to reach out and connect with others who they feel are like them, that they would take the chance to create a user account on a system they know nothing about, that hasn’t been maintained in years.
I won’t take the site down; something in me just likes the idea of people connecting through something so random.
– Morgan Schweers, CyberFOX!
Greetings,
I sat in my office today, listening to a phone interview in which the candidate was asked a number of simple problems, and then the harder design question:
Given two computers linked over a slow link (say dialup), each has a 1TB file, how would you determine (1) if the two are different, and (2) what that difference is?
Me being the gadfly that I am, I spoke with my office-mate a little about what the ‘right’ answer should have. He acknowledged that he was looking for a subdivision answer, which determined the files first difference location. It’s a pretty simple answer, conceptually subdivide the file into (say) 4096 separate 256MB blocks. Do the MD5’s on them, pass them over the wire, so each side knows the MD5’s at each 256MB step. If there’s a change, instantly at that point you know it was in the last 256MB’s, and you can subdivide that down by 4096, so you’re MD5′ing 64K blocks. This finds you the 64K block where the first change happened, and you can either subdivide one more time, or just transmit the 64K containing the difference and figure it out right there.
I pointed out that this only finds the first byte where there is a difference, and that if you wanted to know if some portion of the *rest* of the file is the same (or worse, you want to construct a minimum edit difference), you start to need to do more complex things. I went off, and started thinking about algorithms to make the rest of the problem tractable.
This is what I came up with, recorded here for posterity, and in case anybody with a decent sense of algorithmics would like to correct me.
It’s CPU-intensive, firstly. It’s CPU-intensive, because that’s what you’ve got. You can’t use bandwidth, because it’s defined as too low for this, so you have to use what you’ve got, and that’s the CPU power on each side.
So one machine designates itself as the ‘compare to’ server. It then chunks the *remainder* (point from diff on) of the 1TB file into fixed size chunks. (Say 64K chunks, which in the worst case would be 16 million of them.) Each 64K chunk is separately MD5′ed, and the results stored on disk, hashed by their bottom two bytes into a linked-list, along with the block number they’re for. The bottom two bytes are used as an index into a 64KBit bit-vector, and the bit at that location is set. Once the remainder of the file is completely processed, the bit vector is transmitted.
The ‘compare from’ client then creates (this is the computationally sucky part) 64K-1 simultaneous MD5’s, chunking from (diff+1) to (diff+65535). When it’s done chunking 64K of data (all 64K MD5s will complete at the same time), they’re checked, reset, and they all start again (at diff+65536 to…). This is defined as 64K-1 MD5s under the hope that some kind of internal MD5 optimization could be found that would prevent it from behaving as slowly as a single MD5 run on the order of (64K*(EOF-DiffPoint) times.
During the ‘check’, the bottom two bytes are used as a lookup in the bit vector. If any MD5s match a bit, all matching ones are transmitted, and the server checks if any are a recognized block. If any is, it returns the block number, and the next 64K MD5, which the client can quickly check by running a single MD5 on that data. If they match, the client and server can consider the rest of the file as if it were to be checked from the first step again, because there may be more than one point of difference. More precision can be gained by transmitting the difference data from the client. Since we know there is a difference at the first byte (by the first algorithm), we do a backwards compare from the end of the last different block on the server versus the block before we found an MD5 match. We then find the last byte of difference, as well as the first. (Because of the size of the data, you could transmit it to the server, and run a more normal minimum edit difference algorithm against that block, to get more precision.)
As I said before, treat the rest of the file (from server block ‘N’ and client block DIFF+n bytes to the block ‘N’ match) as a new file to compare. Scale down the blocksizes in this new instance, for quicker/greater precision, and do it again.
This should result in a decent minimum edit difference ‘collection’ on the server (or whichever side aggregates this data), without transmitting the entire 1TB of data over the wire. These numbers can be tuned, e.g. the two-byte bit vector can become 3 bytes, and still only takes 2MB up front to transmit the bit-vector, but substantially cuts down the likelihood of having sub-hash matches, and thus fewer packets sent later.
Now, a real problem comes if 1 byte in every 64K is modified on the client. The problem is that it’ll never match another block. The CPU load becomes pretty excessive at this point, effectively doing 64K MD5s of a 1TB(-64K) file.
At every block size, this problem remains, unfortunately, and with a file as large as 1TB, you really want to start with large block sizes. If, for instance, every 64th byte is altered, you’re pretty much doomed as that’s the blocksize for MD5 and you’ll never get a full match, and at that point in order to get a full edit difference between the two files, you will need to transmit one of them to the other.
So this approach is basically restricted to when you have an idea how many changes (or expect that the answer should be zero) have been made.
The original interview question, however, is even more restricted in that it solely finds the first location of a change, and does not identify what kind of change it is, which is in my book a not terribly useful answer.
– Morgan Schweers, CyberFOX!
Greetings,
In retrospect, it was a Very Good Weekend… It didn’t seem like it would be, given the time it started.
Saturday involved waking up at the inhuman time of 7:30am to get to the Seattle Center - Exhibition Hall by 8:45am, to register for the Magic: the Gathering Ravnica pre-release with , who brought a darn good cinnamon bun which made up for his being on Foxtaur-standard time.
I went 2-1-1 (2 wins, 1 draw, 1 loss) with a red-blue (black splash) deck in my first tournament in over 2 years. (My previous was the Mercadian Masques pre-release, and before that it had been a solid 3 years since my last tournament.) I really like the set, it feels…comfortable, somehow. After that, homeward, to catch up on the sleep I missed out on to get there so early, and some delicious dinner. Shaterri came over, played some online poker on the Monsterous Black Couch, and just idly hung out as we watched bad 80’s videos and all plugged away at our own little projects. (Laptops are amazing devices, acting with a powerful suction to draw the eyes to them. At one point all three of us (my fiance, me, and Shaterri) all were engrossed in our own little laptop worlds. The very model of geek socializing. At least we weren’t using IM to talk…
) Despite this, there’s something really comfortable about just hanging out, odd and interesting ideas and theories popping up every so often, blazing out in a fire of brief conversation, and then settling to a genial silence for a bit more.
We started Sunday with breakfast at the Brown Bag Cafe in Redmond, meeting Shaterri there along with another local acquaintence and a friend/couch-surfer of theirs. I was poked a bit to see if my Big Company had job openings for the friend, but without knowing more about what they could do (and they were surprisingly silent all breakfast), I didn’t know where to point them. Plus it’s not exactly like I’m overwhelmed with happiness about what I’m doing myself, which makes it hard to pump to others. For whatever reason, I can NEVER finish all my food at the Brown Bag Cafe. It’s really Good Stuff(tm), but overwhelming. Highly recommended…although I’m going to not go back for a bit, having had it a bit too much recently. It’s really pleasant to spend time with fellow geeks; the new ideas, knowledge, and pervasive excitement at technology is always invigorating.
A bit of shopping, a bit of driving about to find a store that sold ‘We love Katamari’ and ‘Burnout: Revenge’, and it was homeward… Somewhere along the way my lovely fiance had the idea to have fondue that night…
Being of the cheesy sort myself, I would certainly never turn that down…it was just a matter of getting the appropriate tools and parts.
We went to Target (because it was open) and bought a cheese fondue pot, but missed the part about it needing gel fuel to burn, and that you actually melted the cheese in a separate container (mea culpa, sigh). My brilliant lady solved this with a pair of candles, and a double-boiler setup to melt the cheese… QFC actually sells bags of a couple of different types of bread in nice round slices that when you quarter them are PERFECT for fondue dipping. A small veggie platter later, we were in business.
For future reference, the fondue recipe we used (as I recall it) was:
Mix the sun-dried tomato and beer together and bring the combination to a bare simmer, then drizzle in the cheeses (blue, sharp, mild+mozz), slowly, mixing as you go. Deliver to the fondue pot, sprinkle with a healthy helping of pepper (ground is best, standard flakes work), mix thoroughly, and start your fondue forks!
Some cherry and strawberry tomatoes, raw broccoli, small canned potatoes (each cut in half usually), and apple slices were joined with a bunch of the quartered bread pieces above.
The bread is best if you let it sit for a while, so it gets ever-so-slightly stale. It’s more structurally strong, as well as acquiring the cheese easier.
It was truly, truly tasty, fun, and a great closer to a damn fine weekend.
We put off the typical chocolate fondue dessert for a little while until we can get some flavorings to build a really tasty mixture…
Our fondue ’sensibilities’ come from spending way too much time (and money!) at La Fondue in Saratoga, CA… We adore the Blue Tomato, and the Decadent One dessert. Being about 800 miles away from La Fondue now, this is our best interpretation of the delicious food to be found there.
After the delicious cheese/bread/veggie dinner, a bit of gaming (Burnout: Revenge for me, World of Warcraft for my lady), the weekend, wonderful as it had been, was over. I don’t need many weekends that comfortable, but it’s really, really nice when you get them.
Here’s hoping your weekend was as nice!
– Morgan Schweers, CyberFOX!
content
Subscribe to my feed.Theme: Unearthed-Refried-Fluid
CSS-only (no tables) fluid two-column custom layout by Morgan Schweers!