Awesome, more import and syncing


There's another project on the radar at work and I'm not sure how easy it's going to be to get working.

Any time that you have to keep a concrete 5 website updated with content from a secondary system it seems like there's potential for problems to show up.  I say this really only based on two sites that I've worked on doing any sort of import on. 

The first one is Greaserag, import there turned out to be much more complicated than what we had originally planned on.  Every step of the way there were little gotchas that showed up that didn't work, the biggest were the Oembed auto parsing of Flickr and Youtube and Vimeo urls, then all of the captions had to be imported.  Then I had to come up with a way to split the posts based on an HTML comment instead of just splitting on a number of characters and stripping out the html tags.  I never really liked stripping HTML tags for shortening content on page lists so the splitting on a comment thing has turned out pretty useful. I wish there were more places that I could use it, I'd really have to write my own blogging platform to really take advantage of it.  I've actually thought about this since the ability to import wordpress content and then blog with the some of the same features of wordpress would probably be a pretty well selling add on.  I might do a lot more focus on this over the summer if I don't get back into doing Bikelove.

But even with all of the problems and hurdles that needed to be overcome to bring wordpress content into concrete 5 it was still just one import.  There was no syncing or trying to make it so that you could continue to publish blog posts on the wordpress side after you imported on the concrete 5 side of things.  It was just one fell swoop.

Contrast that with the other site that we've done a lot of import work on and you start to see why syncing data is so hard.  On this site we did concrete 5 ecommerce import syncing to a CSV file generated out by a point of sale system.  That seemed like it would work in theory but there were several things that didn't work on it.  We started out with an add on from the marketplace that claimed to do product syncing but there were several things it didn't do.  The biggest one of these was keeping track of the attributes for size and color.  We had configurable attributes for each one of these options that were required when adding the product to the cart.  When you import the products it would completely wipe out all of the attributes and replace them with new ones.  This means that any reporting being done by the back end on existing orders now has invalid attribute key IDs because the attribute in question no longer exists.  So you get no information about previous orders when you look at the reports, you know which product was ordered but not what size or color.  Not good at all.

That was pretty easy to fix with a custom select attribute type.  What I did was to make each option in the attribute type list have an option for enabled or disabled, then I set that flag based on what's in the CSV file.  This keeps from deleting each key and ensures that out of stock sizes and colors don't show up as available to order.  We eventually got most of the other bugs worked out but even with moving to doing the import and sync from the command line it can take up to an hour or more to run the full product import when you are importing images.  There are just so many different tiny little things that have to be set for each product.  Even then we still run into edge cases where things don't import properly or somehow a person orders something that is out of stock.

So with that as my experience working with importing and syncing data from a concrete 5 website you'd think that I wouldn't want to do any more importing or syncing.  You'd be right.

However, you can't always pick what you are going to work on.  I found out about a new project coming up sometime in the next couple of months at work which is going to mean more import and update.  This time it's going to be users that we have to update, and we also have to keep the remote system update with the information from our users when the user updates their profile.

I'm worried about this for a lot of reasons.  There are a couple fields in the 30 or so entries that they want to track that have to be pulled out of information from the CSV file that the remote system generates.  So that's probably going to mean a custom attribute type or manually updating the options in a select list, I may even have to do something like in the product import where it's a "Persistent Select" and enables / disables registration.

There's also modifications to the registration process that I'm not sure how to work out either.  We're going to need to collect credit card info and send it to and get a recurring billing ID and store that as well. 

Hopefully it's not as bad as I'm thinking it's going to be.  We might do something like hooking into the on_user_login and on_user_update events to pull in data from a secondary table instead of using user attributes, that might make coding it up quite a bit different.  I'm not entirely sure how we're going to be using all of this information that we're going to be collecting.  I think that it's really for stuff on the other server that we're updating, so that they can do the billing correctly.  I don't think most of the fields that we're going to be tracking and updating in the profiles are actually going to be used or displayed on the final site anywhere.

We're still a ways out from coding any of it, the design files are still going through the revision process and the client is apparently kind of dragging their feet a bit so it could be even longer until we see files.  Once the designer has the go-ahead we'll be able to start in on this and maybe see more information about how the user profile fields are going to be stored.

I'd be curious if any other concrete 5 developers have done something like this before, updating files between two different servers to sync user information.  I'm sure that there are probably ways to make it easier that I might not be aware of.

Other than that things are slow today.  There isn't really much that's at a point that I can do it at work today so I'm spinning my wheels a little bit.  I guess that leads to more blogging.  There will probably be more blogging in the future anyway, I'm not sure why but I feel the need to post more often.  I haven't really been great about really posting up what's going on, most of what I do is just based on what I'm doing professionally.  I don't ever post up anything about the stuff that I'm watching or reading or really say much about how I feel politically.  I haven't figured out how to translate that into blog posts, there are a million and one things that I read and share to facebook about how I feel about what's going on in this country but I never bring any of that stuff to this medium. 

I'm taking several days off over the course of May to try and get some more riding in to have some base miles for the Powderhorn 24.  Seems sad that I have to actually burn up vacation time to get miles in but I wasn't finding the time with everything else that I'm doing.  I took two Thursdays and one Wed off so far, I figure anything further out in the month I can think of later.  Maybe I'll be doing better at the distance rides by then and won't really have to take time off to get the miles in.  I plan on doing the one day during the week then trying to get one day on the weekends. The way I have been riding has really just been my commute so 22 miles 3 times a week.  That's not really all that much.  I need to spend a lot more time in the saddle if I'm going to be ready to spend 24 hours doing nothing but riding.

I posted up on facebook looking for people to ride with a couple times but didn't get  a lot of replies.  I guess not many of my friends want to go out on long rides during the week.  Or maybe they just don't want to ride with me.  I've always found that I enjoy riding with a group of people or at least one other person a lot more than I enjoy just riding with myself so it was a little disheartening to post and not really get much of a response.  Still, I found two people willing to ride with me so I won't be stuck alone the entire time.  That's something.

Overall my attitude is pretty poor right now so it's no surprise that nobody wants to hang out with me. I don't see it changing any time soon.  It's been suggested that I need therapy but I don't see how I could afford real therapy and Icarus doesn't really seem to be helping me out much at all. So I really don't know what to do. I guess just learn to live with being depressed all the time.

It's not even that I'm really feeling all that bad but when I do let anything slip that I'm not happy people jump all over me and accuse me of being overly depressing and having a bad attitude  I think I have a pretty realistic attitude given all of the factors involved. My life is boring and small and worthless so there's no chance in anyone wanting to share in my life.  It's a simple as that.  And as long as I'm not happy with my life I'm going to have this same problem.  I'm always going to be unhappy with my life because there's so much room for improvement, nearly everything that I try to do ends up in failure.  People try to get me to focus on the successes but even that doesn't work.  Because what other people see as success I still see as failure, because I know what the intended outcome really was. 

The only way I can see it changing is if somehow something I do does work.  And there's not a lot that I'm doing right now that really has this potential.  Maybe the Occupy site, that could be something that would make me feel like I was doing something worthwhile.  But even there I wouldn't be happy with just doing it for one city, I'd want to set it up for multiple cities and see it change the world.  Then I would consider it a success.  Just getting it working in Minneapolis wouldn't be enough.  And even that I don't know if it would actually make me feel like I'm worth anything.

That's probably because the only time I've ever felt like I had any worth was when I thought that someone loved me.  And that's been a very small percentage of the time I've been alive.  And as long as I'm miserable there's no chance that anyone will love me.  So I'm stuck. And even that doesn't help because I'm always going to be filled with doubts that they actually do love me, because after all, who could?  That was the key to what went wrong with my last lover, I couldn't trust that she still even wanted me with all the time that I was spending alone and it put far too much strain on the relationship and she didn't want to be around me any more. 

So I'm supposed to just be happy with failure and being alone, that supposedly will fix everything.  That's what people keep telling me.  But it's been proven over and over that I'll never be happy.  There is no "Happy" in Jeremy's world.  The best that I can ever hope for is just content and even that is hard.  I'm not content with what I have because I know that I should be doing more.  The code I write will always be just mediocre.  The speed I ride my bikes will always be slow.  The causes I feel strongly about are doomed to failure. The women I find attractive will always have better men to love and never love me.

So yeah, that's about where I'm at today.