InsaneJournal Announcements

I just wanted to update you all and let you know where we are on the site move. We have moved the user databases over to the new servers, but we still have the master database to move. Also ALL userpics are now on the new servers.

Also tonight, I moved the webservers over to the new servers. Unfortunately this led to the side affect of being unable to read some of the cached style data for S1 users. Some of you probably noticed this with 500 Internal Server Errors. This has happened before when we moved webservers, but I was unable to track down exactly why it keeps happening.

The good thing is that I was able to fix the errors. The bad thing is some of your colors or overrides may be missing. I apologize for this, but there is no way to get them back. This is a very unfortunate side effect of moving to the new servers. Some of you who have been around since the initial migration (immediately after Strikethrough) may remember going through this before.

I believe that by moving the webservers and optimizing the databases that we have corrected a lot of the slowness issues with the site. The userpic situation should calm down too as the webservers are now in the same rack as the userpic servers and the site no longer has to fetch the userpics from across the internet.

There is still some work to be done to finish moving/optimizing the site, but we are getting close and hopefully we will be done over the weekend. There will be some downtime when we finally cut over the master database to the new servers, but that should be the last of the downtime.

Now I can actually cancel these old servers and we wont have to worry about the old hosting provider ever again. And just so all of you know, so far I am THRILLED with the new hosting.

On an unrelated note, you may notice my icon (if you are not reading this on the front page of the site). I am running for a seat on the LJ Advisory Board. I feel that the "LJ Clone" sites need a voice in the direction of the LJ code. And if I get on the board and am able to convince them to make some much needed changes, then we just may be able to go and pull updates from LJ once again.

If you want to support my candidacy, there is an asylum here vote_squeaky and a sister community with the same name over at LJ.

If this doesn't work out though, I have some plans, and we will need programmers who believe in FOSS and the Free Software Movement to join us .. keep an eye out!

I have completed moving all of the users to the new database. This means that site performance should begin to pick up.

I am still looking into the errors you are getting while commenting. If you see an error that shows that the database is unavilable when commenting, your comment has, most likely, been posted even though you see the error.

If you are still getting an error that says your journal is in read only mode, please let me know. Also if the error is only happening when posting to an asylum, please let me know what asylum.

Edit: I think I tracked down the "Database not available" error. It had to do with subscriptions. So to resolve this, I temporarily had to run them off. I will look into it more tomorrow evening. Please let me know if you are still seeing these errors on the site.

Here is what happened. When I set up the new databases I made a mistake in the way the database files were laid out. This, combined with the rate we are moving accounts to the new database caused the database servers to get out of sync.

The site protects itself when it gets out of sync by not displaying the out of sync data, hence the "Database not available" error messages you were getting trying to post comments. This was also why you weren't seeing new posts when you visited your journal or your friends' page.

So what I had to do was shut off the site and fixed the problem with the database that was causing it to run slow. Since the database is so large, it took a long time to do this. Now that it is back up I am migrating the final batch of accounts. This is causing the site to run slow right now.

I am in the process of moving accounts to the new database. Some accounts are currently stuck in read only mode. I will resolve this asap.

While your account is moving it gets locked into read only mode so nothing gets lost. Some accounts failed to move so they are still locked. I am going to resolve the reason they are stuck in read only mode and get them moved asap.

Edit: The problem wasn't locking, but one of the new database servers was running slow, I switched to the other and we are doing fine. I have 35k accounts scheduled to move right now. Also moving to the new databases will (hopefully) alleviate the comment rot problem. (For your DB types we are moving for a MyISAM engine to an InnoDB engine). The code handles InnoDB a little differently so hopefully stuff will get committed a littler more regularly.

Edit2: I found an issue with the configuration on the new databases that was causing all the issues with moving journals. This has been resolved. I will have everything out of read only mode soon.

Edit3: If you are still having issues with a journal in readonly mode, please post the username here and I will fix it.

Edit4: Your userpics are still there. They userpic server is just running a little slow due to the migration of accounts. You may want to hold off on uploading userpics until the migration has been completed. I will post once it is.

I am going to be taking down the userpics server for a short period of time as the next step in migrating the site. You will not be able to view, modify, delete, or upload userpics during this time. We have a backup of the userpics database so we will not run the risk of loosing any userpics.

Edit: The userpics move is complete and userpics are back up.

I have starting moving some of the data to the new servers, and it appears that some userpics may have gotten lost in this process. If you have any userpics that are consistently showing up as a broken image (i.e. a refresh doesn't help), then you will need to delete and re-upload them.

I assure everyone that once the move is completed then we will no longer run into issues such as this. The issue occurred because we were using an old and buggy version of the userpic storage system that wasn't properly backing up the userpics. I have upgraded that system and it is running off the new servers so this will not happen again.

I deeply apologize for this issue and once agian assure you that it wont happen again.

Also please be aware that there may be some downtime over the weekend while I move the site. Please bookmark our twitter http://www.twitter.com/insanejournal as I will keep a status update of any downtime posted there. I will also post a notice here giving everyone at least a few hours of notice before any actual downtime.

Just wanted to give you all an update on a couple of site issues. First of all the site move is starting to pick up steam. The hosting company finished building out the new servers yesterday, so I am going to begin the install process immediately. I am hoping to move the site to the new servers this weekend, so keep an eye out for that.

Also, for those of you who see advertising, I am currently trying to clean up the ads, I haven't had the time to do it the past few weeks, and some undesirable ads slipped through. Please have patience with me, I do not want annoying or misleading ads running here on InsaneJournal any more than you do.

If you have any questions for me this is the place, but if you are having an issue with the site, as usual, please open a ticket with our support team at http://www.insanejournal.com/support/

Okay, we have met and slightly exceeded our fundraising goal. Thank you all very much. I am going to end the sale of Permanently Insane Accounts and Insane Userpics tomorrow morning. I want to thank all of you for your generosity,

Also we have started the ball rolling with our new hosting provider. It will take them about 2 weeks to get the servers ready for us, then it will probably take me about another week to get everything moved over. I will try my best to keep everyone updated. Due to feedback I received we are going to go with a more expensive, but much more professional provider for our hosting. This will give us better response times and a much more robust hosting solution.

But this also means we are going to be forced to raise priced of our Self-Committed [paid] accounts. We have decided that we aren't going to raise them as far as we stated, instead we will work on delivering some new products to all of you. The new prices will go into effect tomorrow at the same time we remove the sale of Permanently Insane accounts and Insane Userpics. The price will be 6 months for $15 and 1 year for $25.

Once again I want to thank all of you for your generosity as we move InsaneJournal to the next level of professional quality service.

Edit: The InsaneJournal Store has been updated with the new prices as well as the removal of Permanently Insane Accounts and Insane Userpics. Thank you all once again for this highly successful fundraiser.

We are getting close to having enough funds to setup with a new hosting provider. Right now we are about 65% of the way there. Keep up the donations and purchases from the InsaneJournal Store. Some of you have asked for a donation button, so I am providing one right here

Edit: We are getting closer. We are at about 90% of the goal for fundraising. Please note that once we reach our goal we are going to stop selling perm account and insaneuserpics, so hurry up and get them while you can!

After the longest outage in InsaneJournal history we are back 100%. No data was lost. I will post a larger update later explaining what happened. For now I simply apologize for the issues.

In the future if you want to keep up with what is going on during outages such as this check out the InsaneJournal twitter at http://www.twitter.com/insanejournal

Edit: The random missing page errors should be gone now. Sorry about that

Edit: AOL is currently blocking InsaneJournal mail. We are working with AOL to have this matter resolved asap

Edit: It appears that all notifications and emails are working normally again.

I had mentioned in my previous post about a site fundraiser. We have decided to go forward with this. I was planning on offering vgifts and extra userpics for paid users, but was unable to get the code finished in time. Don't worry though, those items are coming as soon as I can get them ready.

So you know the reason we are running this fundraiser is to handle the costs of moving the site to a hosting provider that will better suit our needs. Our current web host recently charged us an overage charge that managed to double our single month costs for running InsaneJournal. We need to move ASAP to keep this from happening agian in the future.

We are going to be running this sale until we can raise the costs needed to cover setup and canceling the contracts with our current provider on two of our servers.

Please don't see this as a sign that the site is running out of money because we are not, we just don't want to drain our reserves for the cost of moving the site.

Now for how you can help. We are starting off by offering Permanently Insane accounts from $60 and InsaneUserpics (5000) userpic slots for $175. You can purchase these items by going to the InsaneJournal Store.

If you want to help out a friend and help the site at the same time, here is a way to find out which of your friends have or don't have paid accounts http://www.insanejournal.com/paidaccounts/friends.bml

Once we can get the site moved and stable I can start concentrating on squashing bugs again. Please don't feel that I have forgotten about your issues because I haven't. I have just been consumed by running the business side of the business as well as keeping the site up and available all that I can.

Overnight last night we hit 200,000 accounts. The growth on this site is amazing. It took over 6 years to reach 100,000 and then less than 3 months to double that.

You guys rock!

I just want to post here and apologize for the recent slowness some of you have been experiencing. We are aware of this issue and its cause. Our primary load balancer went over its monthly bandwidth allotment so we had to reconfigure the way the site runs. I was hoping that it wouldn't be noticeable, but alas, it is running very slow at times.

Unfortunately there isn't much we can do about this until the billing cycle starts over in about a week. So until then I am going to have to just let you know that the site will be running slow.

We are already taking steps to keep this from happening again. So after we get things going again, we wont run into this problem in the future.

Also we might be switching hosting providers in the future due to this issue. And if we do we will be doing a fund raising drive on the site. We haven't worked out details yet, but we will let you know well ahead of time.

I want to move this subject back to the top of the announcements asylum, so I am reposting it here.

The new version of the InsaneJournal code is up and running. Please report any major bugs here. And before reporting a bug as new read the comments and if it has already been reported, just comment to the report that you are also experiencing the bug.

Here are the list of known fixed bug and known bugs that are on my list to be fixed.

Fixed Bugs:
* Interests Search is_visible error
* Tag Editor not working
* Edit Journal page not styled properly
* Admin Console not working
* TOS Requirement not working properly
* Very slow email notifications
* Approving moderated posts
* Opening support request using beta scheme causes site to error out
* Renaming an account to another account you already own isn't working
* directory.bml unable to load without an error

Known Bugs to be fixed:
* directory.bml hangs searching
* Comments still seem to be disappearing.
* Quickreply to comments not working
* Edit userpics page loading slowly for users with lots of userpics (Firefox is still slow but wont timeout and give an error as a temporary workaround)
* Adding friends via the admin console causes custom groups to get emptied
* Joining a closed membership asylum
* Timezones on update page not reflecting change in DST in United Stats
* align= and border= options in image tags being striped by the html cleaner

We had a short outage tonight, thanks to a couple of loyal volunteers alerting me to it being down. I found the problem was that we ran out of memory on all of our webservers. It seems that the new code uses more memory and we hit a really busy peak this evening.

I have made a change to the webservers that should let them keep chugging along and everything should be responsive for the rest of the night.