What happened: 2016 edition.

[Admin]

So, the DB decided to go to Vegas for a couple days. Which is interesting because I was unaware that MySQL had any idea what Vegas was and the fact it's supposed to be a relational database would preclude something like gambling, but ok. It can do whatever it wants I suppose.

The host here fucked up, again. I will not mention them by name as I've read horror stories of accounts getting shut down for "Technical reasons" online so please don't mention them by name either. This is the second time this year the database connection just vaporized due to some bullshit reason I no longer care to listen to. What should have been a 30 minute "fix" turned out to take the better part of two days. This is unacceptable.

Every time this happens I talk about moving to a different host and then don't because things are ok again. Then it hit me... That is the honeymoon phase after the beatings.

So PL needs to move out. It's already resource constrained here and the amount of hackish shit I have to do to make it work is really not how things should be run. I don't even like thinking about the fact that emails are still not being delivered because the host has a special way of getting their core mail server in multiple spamhaus blacklists and all attachments are currently on the file system because the default storage method of keeping them in a database causes it to keel over.

At the turn of the year they removed root access to their VPS platform clients(me included) with only a 2 week notice and then proceeded to destroy all the files owned by the user with sudo access. Glad I had backups. Speaking of backups, one of the services they are supposed to offer is automated off site backups, but, per normal, they skip users that have more than, get this, 4GB of data... Attachments here are 25GB, FTP is 60GB, Hell, the text/post only database almost half of that. But it's ok, because the host has a solution! For even more money I can move to their unmanaged service to fix a problem they created. This does nothing to address the MySQL or email issues. This is, frankly, unacceptable.

My goal is to get PL in a colo environment just so this never happens again and the random slowdown/hangs/crashes/database explosions/email problems just go away. And if they show up, It's either my fault or something goofy with user email providers and I'll have bare metal to punch soundly about the face to get it to work again.

Which brings me to my next point: The donation widget thing in the top right I publicly displayed. I put it up because subscribers were under what it takes to run this place a month. Since then there have been 2 new subs and PL is revenue neutral again. This is good but not sustainable given the trend is down year after year.

PL needs hardware and a data center to live in. These things are thankfully cheaper than they used to be and the hosting cost per month will go down after the initial costs are taken care of, even after incremental offsite backups covering all data. Moving forward I need to purchase a license for Vbulletin5 because they no longer fix or back port fixes to Vbulletin4. This includes security fixes. Mixedgas and a few others have been vocal about the text formatting bugs in IE10. It's related to the embedded editor and has been fixed in VB5 for a while. License is $200 and change for an upgrade. All this junk adds up.

But I hear ya, why don't I just take care of it? I can't. Money I make goes to taking care of bills and living these days. At this rate, using PL subscriber revenue to do this will take the better part of forever. Maybe a year if I put up ads everywhere, which I almost had to do before putting up the donate link in the top right.

Bradfo69 said "There is no "PL" company. It's a regular geek hobbyist guy like the rest of us, doing this out of the basement on very little coin and has been maintaining it 24/7 for many, many years now. That's a tremendous amount of dedication." and this hit me right in the feels. It's true. I started this place back on January 16, 2005. The idea this place has been running over a decade didn't really sink in until I realized that's a solid 1/3 of my life. I guess I get introspective while staring at a terminal window watching data shuffle around. I don't run it like a business, I'm just a guy who made the same expensive mistake of getting into lasers like a lot of you. I also made the mistake of starting a totally awesome forum. And even though I don't post all that much, I really like this place and the amazing people floating around, and want to see it continue going another decade or so.

So, after this happens, here's what I expect everything to look like:

Dedicated mail server:
Will handle the output of the board and provide provisions to give Benefactors email addresses at PL.
Dedicated Apache server with PHP:
This will house the forum, the sorely missed Wiki, and any future services.
Dedicated MySql server:
This will provide the data to the Apache server, store, replicate, and backup the data of the above services.
Dedicated FTP/SFTP server:
This will house the FTP contents and provide credential authorization to upload and retrieve files.
Dedicated IRCD:
Live Chat system
Offsite Rsync/ MySQL repository:
Daily incremental backups to a secure location that can be recovered from in the event of a catastrophic outage.

All systems with the exception of the Offsite storage are expected to be SATA SSD based RAID1 or RAID6 volumes.

Some services can be combined into a single server, such as IRC and FTP, or Apache/PHP and MySQL but given their different hardware requirements and propensity to be exploited a logical separation is considered best practice.

For example:
FTP requires slower bulk storage - Large storage size disks, typically TB+ rotational disks though recent SSD developments have changed this a bit even if cost prohibitive.

MySQL requires as much raw I/O as can be provided - This is, topically, best provided by a SSD array

Mail servers are notoriously processor heavy and don't play well with themselves, much less anything else.

Expected downtime for the changeover is about 2 days give or take. 12 hours for moving the data around and about 2 days for DNS propagation to happen.

Anywho, that's my thought process and I will absolutely need your help to get it done.

Feel free to comment below and let's get a discussion going.

[Admin]

Reserved for future edits/additions.

Thread promoted to a global notice.

[Aidan]

Hi Spec,

Thanks for all your work over the years, PL is a wonderful bounty of information I hope will live forever.

Looking at what you said above it may seem a little OTT to split everything across multiple box`s as resource wise the whole site should be able to run on one box with no issues at all and the cost of many boxs isnt ideal capex or opex wise.

If you want to split them up at an OS level then visualization would make sense, also massively portable should there ever be any issues, in fact IMHO it should be virtual regardless.

I have many contacts for data centers here in the UK, I live up the road from one and funnily enough im about to pull a decent box out next week.

I expect you wouldn't want to move out of US so maybe a load balanced EU mirror, lots of possibilities.

Let me know how much the hosting is costing and what your planning, I would be surprised if I cant help in some way....

[Admin]

Yep, PL certainly doesn't need half a dozen specialized dedicated boxes and I'm toying with the idea of tearing everything apart into esxi instances on a system or two and have a doc here on my desktop going over the hardening process for that. All I remember are horror stories about having the esxi management host publicly accessible. This, as you know, necessitates another host to tunnel though if I want to manage it sanely instead of having to head to a DC every time I want to change something. The other option is living somewhere in Amazon's cloud. This, interestingly enough, is surprisingly expensive.

Thank you for your offer and I will keep it in mind however I'm currently stateside, and would prefer to keep things "a drive away" if shit hits the fan.

[Aidan]

Yep, that makes sense.

Happy to look at a load balanced Europe mirror if needed at some point.

I've ran esxi for donkeys years in both private cloud and public facing scenarios, there is no real risk these days as long as you keep up to date and don't use "Password1".

If worried use a firewall also, although that may be paranoia!

Restoring vms is a lot easier as well, give me a shout and I can help if needed on the VMware/veeam backup etc setups.

Not sure what you use now MSN gone but I'm on Skype Or FB IM.

[lasersbee]

Thank for the heads up and letting us know what had transpired.
Good luck with the PL upgrades.

BTW it would help if you allowed one time Lump Sum payments as
well as recurring. I'm sure more members would donate one time
payments more often.

Not crazy having another monthly bill to keep track of. Trying to get
rid of as many monthly payments as possible as I'm reaching retirement

Jerry

[absolom7691]

Suggestion. Tier the sub levels more. It's my understanding that there are only two,hobby level and corporate level subs. I would say maybe 3 or 4 different levels at which people can pay. Not sure if there are any incentives to offer a higher level subscription but just a thought. Either that or bump the basic hobby sub to $15/month. I can't speak for everyone but I don't think many would mind paying it.

[Admin]

Thank for the heads up and letting us know what had transpired.
Good luck with the PL upgrades.

BTW it would help if you allowed one time Lump Sum payments as
well as recurring. I'm sure more members would donate one time
payments more often.

Just glad that it's over. There are few things less satisfying than bashing ones head against a "support" wall and waiting.

I want to enable the one time lump payment thing, and have been meaning to for a while but vanilla vbulletin doesn't support it. There are a couple for pay mods that will shoehorn this in and I'll be looking at them over the next couple of days to see if anything has changed.

The other option is setting up a new class of benefactor with hard numbers tied to an amount/time, but even this has issues because if someone that's already blue'd up decides to use it - the shorter time frame becomes authoritative.

Suggestion. Tier the sub levels more. It's my understanding that there are only two,hobby level and corporate level subs. I would say maybe 3 or 4 different levels at which people can pay. Not sure if there are any incentives to offer a higher level subscription but just a thought. Either that or bump the basic hobby sub to $15/month. I can't speak for everyone but I don't think many would mind paying it.

The tiering thing is complicated as I already have this place setup so that default registered users get a couple thousand PM's and no artificial limitations. Most boards I've seen have this value set to something offensive like 10 to incentivize donations. I suppose this happens when one doesn't setup a board from the get go to wrench money out of people.

I really don't want to increase the base subscriber amount. Technical limitations aside(changing that value would break the already broken subscription system) in this day and age of "everything as a service" - even $10 a month for a forum seems out of line. I can only assume places like reddit were able to put in the market research and determine the optimal conversion values based on a given volume of users, and I suspect $10/m for whatever was laughed right out of the board room.

This is, to borrow a phrase from some of the european members here, a sticky wicket.

Side note: See the orange around my name? That's the same thing a corporate benefactor would have, if there were any.

[Datsurb]

The thought of setting up separate instances is not bad. Most apps, have special needs, and will perform best as standalone servers. As Aidan says, using VMWare and ESXi will be the best and cheapest way of doing just that. From what i can see, all the software will be free (opensource) to use exept the vBulletin. That said, free SW is not the same as no cost setting things up and running them. There are still need for a piece of iron, the server. Prices have dropped, but it is till expensive with disks, memory and all the other small pieces needed for getting ut up and running. Then the housing of the server inkl power and cooling. Electricity costs money.

*******
Admin uses his personal time, and does not charge for this. I know that running a rig like PL, takes time, and if Admin would charge "normal rates" for his services, we all had to pay quite a bit to air PL. Admin also needs to put his share of hours getting up to date with IT technology. Running PL locally on a "private rig" require to master a lot of different areas of the IT trade. Network - virtualization - storage - server os - apps - backup. All areas who professionally are served by different people.
*******

Publishing towards internet is not free of charge either. I don`t know what capacity PL have for the lines today, but the responsetime to Norway is OK. Opening attachements like pictures might take some time, but the text runs painless. Some security sw should be used, and there are opensource free of charge to be found, meeting the needs. Suggesting using clam AV and monowall as firewall. Monowall should be found as VMware ready image, so it might be mounted and used at once.

Just trying to put some words en what it takes to air PL. Today we all seem to take content on the net for granted, but there are real people doing real work, and it costs real money to be there.

Respect to admin given.

[masterpj]

Thanks for the headsup. Glad it's all up and running Spec!