> 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd.
Also, test that it's properly disabled with something like `ssh -v yourserver : 2>&1 | grep continue`, because there are a surprising number of ways for that to go wrong (did you know that sshd lets you Include multiple config files together in a way that can override the main one? I know that now.)
I think doing the opposite is better usually, never making changes to the main /etc/ssh/sshd_config and always add your own customizations to /etc/ssh/sshd_config.d/; that way you have clear separation of your own and distro configs, which makes life easier especially when upgrading sshd. Although I'll readily admit that I don't myself follow that advise all the time.
In situations where I have more than one box, one (usually a little embedded SoC) gets openbsd on it, and is used as a bastion host, for exactly the reasons outlined here.
The problems I hit with using Linux for this were different ten years ago, but, based on this thread, things got worse on that side of the fence.
Get something like a PiKVM and drop all of the stuff about being very local since you can find a cheaper overall provider elsewhere and use smart hands once a year either for free or still cheaper than picking the local place you can drive to. Even if you do the things in this guide perfectly it'll break/hang/get misconfigured at some point and the PiKVM (or like) lets you remotely hard boot a box instantly without having to drive or open a ticket. It also enables you to easily reinstall the entire OS remotely if you need to.
If your server/device has an IPMI... get a PiKVM (or like) anyways. Not only will you last more than 2 seconds without being hacked but it'll have more functionality and be much faster.
If you're in the US there are lots of places in the Kansas City area that have ridiculously cheap pricing and it's decently centrally located in the country.
Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes build their own). Bonus points if they offer KVM - as in remote console, not the Linux hypervisor. Also ask if they do hardware monitoring and proactively replace the failed parts. All of this is still way cheaper than cloud. Usually with unmetered networking.
Way less hassle. They'll even take your existing stuff and put it into the same rack with the rented hardware.
The difference from cloud, apart from the price, is mainly that they have a sales rep instead of an API. And getting a server may take a from few hours to a few days. But in the end you get the same SSH login details you would get from a cloud provider.
Or, if you really want to just collocate your boxes, the providers offer "remote hands" service, so you can have geo-redundancy or just choose a better deal instead of one that's physically close to your place.
This. I used to colo lots of stuff, but now mostly use Hetzner. But there are many in this space, and some of them even offer an API. And some of them (like Hetzner) also offer at least basic cloud services, so you can mix and match (which allows for even steeper cost cuts - instead of loading your dedicated hardware to 60% or whatever you're comfortable with to have headroom, you can load it higher and scale into their cloud offering to handle spikes).
The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)
One hurdle that many companies who have only known cloud hosting will face here is significant: how do you find a reliable, trustworthy datacenter? One who actually monitors the hardware and also has a real human if your networking access gets screwed or if you need a critical component swapped at 2 am on a Saturday.
I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone.
Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.
> how do you find a reliable, trustworthy datacenter?
drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself
That’s unfortunate. I consider SoC 2 compliance as a negative indicator of security (I’ve been on the vendor side of it, and have seen it have significant negative impacts on security and reliability in multiple organizations).
Ideally, there’d be locked doors, and the data center wouldn’t be subsidizing performative checkboxing.
I think if you want to host in Europe then Hetzner is the clear choice. They won't monitor your hardware for you though, you need to let them know if something breaks and they'll replace it very quickly.
You're right you need to find a company you can trust.
And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).
I often recommend a "embrace, extend, extinguish" approach to AWS: Starting there for simplicity is fine, then "wrap" anything bandwidth intensive with caches elsewhere (every 1TB in egress from AWS will pay for a fleet of Hetzner instances with 5TB included, or one or more dedicated servers).
Gradually shift workloads, leaving anything requiring super-high durability last (optionally keeping S3, or competitors, as a backup storage option) as getting durability right is one of the more difficult things to get confidence in and most dangerous ones to get wrong.
Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.
> Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.
Every time I contemplate increasing my use of S3 or similar cloud services, I get annoyed at the extremely sorry state of anything on-prem that replicates to S3.
Heck, even confirming that one has uploaded one's files correctly is difficult. You can extract MD5 (!) signatures from most object storage systems, but only if you don't use whatever that particular system calls a multipart upload. You can often get CRC32 (gee thanks). With AWS, but not most competing systems, you can do a single-part upload and opt into "object integrity" and get a real hash in the inventory. You cannot ask for a hash after the fact.
I understand why computing a conventional cryptographically-secure has is challenging in a multipart upload (but that actually all that bad). But would it kill the providers to have first-class support for something like BLAKE3? BLAKE3 is a tree hash: one can separately hash multiple parts (with a priori known offsets, but that's fine for most APIs but maybe not Google's as is), assemble them into a whole in logarithmic time and memory, and end up with a hash that actually matches what b3sum would have output on the whole file. And one could even store some metadata and thereby allow downloading part of a large file and proving that one got the right data. (And AWS could even charge for that!)
But no, verifying the contents of one's cloud object storage bucket actually sucks, and it's very hard to be resistant to errors that occur at upload time.
I usually just use a suitable Nginx config unless there's a compelling reason. It means you "waste" the first read - you just let post/put etc. hit S3, and just case reads, but it's easier to get right. It's rare this matters (if your reads are rare enough relative to writes that avoiding the cost of the first read matters, odds are the savings from this are going to be questionable anyway - the big benefit here comes when reads dominate massively)
Well, S3 is hard to beat for our use case. We make a heavy use of their various tires, we store a somewhat large amount of data but only a minor part ever goes out.
The compute and network heavy stuff we do is still out of AWS.
That's pretty much the one situation where they're competitive, so sounds very reasonable. Some of their competitors (Cloudflare, Backblaze) might compete, but the biggest cost issue with S3 by far is the egress so if not much goes out it might still be best for you to stay there.
Sounds like (unlike most people who use AWS) you've done your homework. It's great to see. I've used AWS a lot, and will again, because it's often convenient, but so often I see people doing it uncritically without modeling their costs even as it skyrockets with scale.
S3 has plenty of competition. It can be a fair price if you rarely read and need it's extreme durability, but that leaves plenty of uses it's totally wrong for.
As a technical person working at datacenter and at still handle technical support requests at some capacity, interesting to read this stuff from the customer perspective. Good to know what is considered and important by customers. Maybe sales staff knows all that too well, but for me just invokes a smile and some pride in the job I do :)
Back in the late 90s/early 00s when I was a precocious teenager, I ran a somewhat popular website. At some point it made sense to just buy a 1U rack moubtable server and having it colocated (commercial webhosting was expensive then). I couldn't find anyone to give me a ride to the datacenter, so I took a bus. By the time I got there my arms were numb from carrying the bloody thing.
There was a single security guard, I signed in and he gave me directions and a big keychain. The keys opened most of the rooms and most of the cages around the racks. To this day I remain mystified at the level of trust (or nonchalance) that security guard had in a spotty teenager.
Back in the early 2000s I had a job that required me to enter a certain phone companies' datacenter. But, I couldn't, because I was a consultant and it was an employees-only area. I had permission to make changes to machines in a rack but they didn't allow me to enter. There was a guard who would check badges but who didn't really check if he recognized people, and there were some people who had to enter and leave multipel times in a day. Myself, I couldn't get permission to enter via the correct channels.
With my director's unofficial approval I was allowed to _try_ to enter the datacenter. So I just walked very confidently towards the entrance, nodded to the security guard like all of the regulars who didn't bother showing their badges, and he let me in.
Having run hosting companies from the mid 90s as well, from memory this kind of thing was pretty normal, even in allegedly secure places like London Telehouse.
Quite a few of us in that era were juggling it with being students, so it wouldn't surprise me if the security staff were used to it and expected you to look young enough to be their kid!
To this day I can get into pretty much any rack or room I feel like at datacenters everyone here has heard of. It just takes experience these days and a bit of charm. Plus having a million keys and staff rack combination codes doesn't hurt. These were freely given and simply added to my collection over time, nothing stolen or social engineered.
I've never done anything nefarious with these abilities, and no one I know has either. It's simply a matter of practicality when you staff a 150,000 square foot facility with 2 security guards who have no idea what they are doing.
If I (and many others) had wanted to, we could have caused multi-week/month outages you'd be reading about on the news with 5 minutes of effort. This is basically the status quo for any sensitive industry.
The world turns because 99.9999% of people want to give you a hug vs. hit you. Society falls when that ratio goes much lower.
That won't work everywhere. I've been to a datacenter in London around 2010 where the entry was similar to the automated airport passport booths. The doors would not open if you were not registered for a visit and there were no visible guards (I hope someone was around in case you got trapped...) I wanna say it was Telehouse West, but my memory is not great.
" 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd. If you just locked yourself out of sshd because you didn't install ssh keys first, STOP HERE. You are not ready for this. "
If you blindly followed the directions and got locked out, you can do exactly the same thing with other directions. You were not ready.
Related to this, is the advice I give everyone for editing the sudoers file:
First, open two superuser terminals. The second one is so if you fuck up the sudoers format so it doesn't parse and you accidentally 'exit' one too many times in the first terminal.
The visudo command is often times available as well. It won’t let you save a malformed sudoers file. Finding other footguns though is an exercise for the reader.
And I find this to be mean-spirited ("snarky") and the opposite of instructive. Perhaps I am new, or rusty, and I do not know about SSH keys or remember them. Bang! Trapped! And then how is a newbie to trust the rest of this, much less learn?
Good instruction will tell you why and the consequences of missing a step. Perhaps some options.
But perhaps I want too much, the title does include the word "terrible," after all.
If you are new or rusty, Stop. You are not ready for this.
Seriously. Not knowing or remembering abut ssh keys is way below the minimum baseline of skills that are needed to manage a server connected to the internet.
Managing colo-ed servers when you're only just experienced enough to know about and know how to set up ssh keys without being told _is_ a terrible idea.
Sorry, but you're still well into the "don't know what you don't know" stage of learning, which is _not_ the time to become responsible for a colocated server.
Go home, from the colocation idea.
Start with way less expensive VMs from your choice of hyperscaler or el-cheapo-vms.com, or in a homelab with old hardware or RasPi/IntelNUC type devices.
If you "need" colocation, you need someone who knows what they're doing to set it up and maintain it properly.
Or just go ahead and wing it. Maybe I'm wrong. Maybe Rachel's wrong. Maybe you're "The One".
Incorrect. Wildly incorrect. I used to co-lo servers, it has been a long time. SSH Keys were not in vogue back then. I did use them at one point and, because I haven't done it in a while, it slipped my mind. I'm just rusty.
There's just nothing wrong with adding a few steps to do your keygen, upload your public key, save your private key on a thumb drive or whatever, and so on. Please, tell me why that should NOT be added to a tutorial. Whom does it harm? What is the downside?
I can't see anything but Crappy Gatekeeping, but please, come up with a decent reason not to add just a few more instructions.
(Although now, apparently, keys are out for some folksand certs are coming in vogue. Who knew?)
How many _other_ important things have "slipped your mind"?
This isn't about ssh keys specifically, it's using ssh keys as a canary - an example of a thing someone who has internalised enough Linux sysadmin and security knowledge will automatically get right. It's shorthand for all the other hundreds of things that need doing to properly secure a public internet connected Linux machine.
This is a way bigger thing that adding a few lines explaining ssh keys.
Perhaps firewall config has also "slipped you mind"? Or TLS certs and renewals? Or shutting down unneeded service started by your distro? Or managing logfiles? Or monitoring disk space/memory usage/cpu usage? Or keeping your JVM/Nodejs/Python/PHP/whatever up to date with security patches? Or maintaining your software BOM and all the security update notification channels for everything you're running whether you installed it ir whether it got bundled in as a dependancy of something else you installed? (Think zx or Log4J)
Or maybe's since you're just "rusty", you're looking for all the sysvinit files you remember being important, not realising your chosen distro now uses systemd. Or your previous experience was on machines from before the era of speculative execution attacks, or from when it was considered acceptable to hash passwords using crypt or MD5?
I don't know when you were doing colo servers while ssh keys "were not in vogue". In the late 90s I was flying annual round the world trips to visit our colo facilities in London, New York, and San Francisco - with half a dozen hard drives in my luggage because replacing raid1 arrays was a better/cheaper solution for us than uploading over our available bandwidth over 56k modems or ADSL. All those machines had ssh keys. It was standard practice.
But yeah, maybe you're right and my 30 years experience and hard earned advice is just "gatekeeping". Same with Rachel the article author's decades of experience (at way higher levels than mine). I'm sure that your assertions of it being "Incorrect. Wildly incorrect" is not another one of the things that've slipped your mind or that you're a bit rusty on. Feel free to assume that having ssh keys "slip you mind" doesn't mean you have a lot to learn (or relearn) before being capable of securely/professionally managing a colo-ed server.
The great thing about having unfettered physical access to hardware is that you can easily recover from mistakes like this. No need to rebuild that EC2 instance. No need to beg a hosting company for IPMI access. You can just pull the plug and try again as if were your own PC.
On the other hand, it's nice to be able to take a no-reboot AMI of the machine you're futzing with, risk breaking it, then just re provision from that AMI if you fucked up. And if you're chasing three or four nines of uptime you do the same but provision a new instance and futz with that one while your working one keeps on keeping on. Even big instances are cheap enough by the hour that you can afford to run and extra one while testing potentially breaking changes.
I faced the same problem, until I found a non-profit community-operated housing provider (a registered computer club/society that has been operating since at least the early 2000 who is also a LIR (https://en.wikipedia.org/wiki/Regional_Internet_registry#Loc...), and they give shelter, bandwidth and IP address space to anyone who participates in covering their operational costs.
I gladly do, and it's the best hosting experience I've had so far, and I used to have rented dedicated iron from Server4You, Hetzner, Webtropia and the like from 2005 on. Maybe there's a similar hidden gem in the area you live in, and you just do not know about it? :) Mine flew under my radar for nearly 20 years, and even though I knew they existed, I was not aware they'd colo other peoples' boxen at very fair rates.
> finding colocation space at reasonable price is difficult these days.
The author of the post seems to be living in the bay area. It's easy when the number of nerds per km² is high, disposable income is even higher, and driving a car for 2 hours is considered "next door".
I think for losers like myself, living in the middle of nowhere with low disposable income, the best solution is just to rent out a dedicated Kimsufi box (from OVH), or a server off the Heztner auctions. (Or whatever is the equivalent in North America) It's much much more cost effective than collocating.
> the bare minimum required to run your own hardware in a colocation environment
i remember the look in the admin's eyes when they asked "alright, what kind of hardware are you looking to install?" and I said "oh i have it right here" and pulled two Intel NUCs out of my backpack
> Consider bringing a screwdriver and a flashlight (any halfway decent place will provide those for you, but you never know).
two multitools minimum, sometimes you need to hold a nut with one while loosening the bolt with the other
the best space is the one that is right next to a Frys/Microcenter/whathaveyou
> "oh i have it right here" and pulled two Intel NUCs out of my backpack
NUCs would be like nirvana after some of the jerry-rigged crap I've seen dragged into facilities. Maybe I'd like them to have a secondary power supply, but then again you had two, but they'd make a fantastic little router for something not too traffic heavy. Lots of utility use cases for something like that in a proper facility.
Don't forget hearing protection! All the fans in there make it crazy loud. Facilities often supply earplugs, but consider using it with over-the-ear protection.
Most server racks will have C13 or C19 outlets for power distribution. For consumer-type devices designed to plug into a wall socket, suitable cables or an adapter strip would probably be required.
A working BMC is great, but I sure wouldn't want it facing the open web. The colo would either need some sort of private network, or you'd need to add a router and firewall of some sort to lock down access to the BMC.
Missing step -1: find someone to get on your staff that knows what they are doing, or at least find a contractor with a good history of doing this kind of thing. After a certain point, you really shouldn't wing it. But if you're at that point, you should know if you actually need your own hardware, or if public cloud or other options are good for you. Some need their own, most don't. There's a lot of nuance there that is difficult to express in a short HN comment.
Agreed. It blows my mind a bit that there is a generation of folks that only know Cloud vs Colo. Colo is a walk in the park compared to cloud, assuming you've done basic PC building and networking. Heck, Cloud can be a walk in the park too for most use cases. If you don't have time to learn it yourself over a month or two, there should be folks available to consult. A lot can be brainstormed in an hour.
Ipmi is a bit of a double-edged sword. Network-connected access to a serial console (including uefi/bios console redirection) and the reset button can be a total lifesaver, I agree. I wouldn't want to be without a serial console and remote reset button either.
But ipmi cards are little network-attached linux boxes which run prehistoric kernels and userspace, exposing as many services as the half-wits who put together the firmware image can shovel in, and are rarely if ever patched by the vendor unless there's some really public scandal.
The standard thing to do is to isolate them on some kind of private management network in an attempt to shield the wider internet from the full majesty of the firmware engineers' dazzling skills, but that might be harder to do in the simple 'beginner' scenario Rachel describes.
One good simple version when you get up to two servers instead of just one is to cross-connect so each machine has ipmi access to the other, but neither ipmi service is exposed to the wider world.
Even if your server does not have proper IPMI, most colo providers have an assortment of ip-kvms, you can ask for those and access your server from any modern browser.
I'm pretty suspicious of trusting the firmware on things like that either, to be honest. (Personally, I tend to use serial cables and a little custom board to turn serial breaks into resets, but I know my NIH instinct is probably a little on the high side. It would be unreasonable for me to suggest someone new to colo start making random extra boards to stick inside their machines!)
You do not need to stick it inside, it plugs into monitor and keyboard ports. Power-on and reset are usually done manually on request. And you do not need it plugged in all the time, only to reconfigure network or boot values.
That's definitely a level of remote-hands intervention I've not had from the handful of places I have racks of colo kit. You're right, it might be something some of them do for other customers though; I've never really asked. They'll certainly push a reset button, swap a drive or rack a server that's shipped to them. (Though hardware's reliable enough nowadays and I have go to data centres so rarely that it's a bit of a fun novelty, and I like doing it myself.)
I've been colo'ing forever. I'd consider a Lantronix spider ipkvm available on request to be like minimum viable colo service. It's usually up in 5 minutes when I submit a ticket to the NOC.
I bet your provider has something like that. It's a godsend when you screw up, say, the hypervisor running state somehow and need bare metal access to unbork it :)
My version of this is to use a serial console server for direct access, rather than vga + keyboard. If the kernel is still running fine but network access is down, I have a getty running on the serial port. If the kernel has locked up too, I hit the reset button by sending a 500ms+ serial break (special circuit, highly recommended) and the bios has serial redirect, so I can do stuff like hit del to drop into bios setup or uefi shell at that point.
But I expect you're probably right some or all of the providers we use do have something like that, as I speculated in the previous post. I've just never understood the point of vga-type stuff when bios/uefi serial redirect exists and serial console is more convenient anyway once the kernel has started, so never asked the question.
Yeah, I like the idea of a bmc but the implementation sort of sucks. What I want: a small soc pony engine that controls the main computer. On this soc I want to install my own os. I like openbsd so that would be my ideal os, but the point is it should be my os not the vendors.
It is indeed very helpful when it's needed - but most IPMI is horrifically insecure, so usually it's not connected directly to the public internet. Instead it's on a separate network with special VPN access.
As this document focuses on keeping things simple, they don't have the isolated network / VPN needed to use IPMI.
What's the importance of having a switch/hub? Is it because you may want to add more servers later, but the colo host only provides one port in their router?
So you can plug your laptop into it at the colo when you set it up so you can verify that the server is working right and can get to the internet while you are still there. And if it isn't working right, it'll be a lot easier to keep everything plugged into the switch, rather than swapping between your laptop and the colo's network every time you try something to fix it.
But yes, if you think you might want to add other machines later, preemptively putting a switch in there will make it tons easier.
It is also useful to configure mutiple IPs on the same interface, one for Colo, and one for your lab. You can do this on all OSes, even on windows.
And if you failed to do this and all advices in the article: then kindly ask colo provider to attach ip-KVM (if your "server" is not actual server with ipmi/bmc), 99.9% of places have these. And most of them will provide it free of charge for limited amount of time.
Documenting and testing the hard reset/reboot procedure, as well as any expectations, for your colocated gear sounds like a good thing to add to this list.
I selfhost a bunch of personal stuff at home and have 1500M/940M at home, and I don’t think bandwidth would ever be the issue.
Frequent IP changes and the residential IP block could make mail hard.
If I was looking to host something that mattered, and was a little more public, I would move that more for my own sanity. Not being able to find me via IP, network uptime guarantees, more robust backup power, etc.
If you were some cloud admin and put your website on a resume or something, and it was down, I might use that to jump to conclusions.
Imho, the easiest way is to get a Mikrotik switch, set up Wireguard on it, allow only Wireguard in, then use the Wireguard Mac App. Build your machines with AsRock Rack or Gigabyte motherboards. If you have existing, then add a PiKVM. You'll get an IPv4 with a half-cab in most DCs.
I suggest a very different reasoning: we know WFH works, we also agree that in large part of the world we have enough sunlight for p.v., battery storage could be cheap [1] and FTTH connections start be be very good for modern time. Who have already understood where I want to go?
Well, we could run decentralized systems at home for SMEs/Startup common needs from modern homes of remote workers, just renting them a room with proper local equipment the modern remote worker probably already have (VMC, A/C, "the big UPS" from domestic p.v.) with not much more expenses than using someone else computer also named "the cloud". Then if/when needs mount some sheds with p.v. + local storage and good connections can pops up. Who need a CDN in such setup? Then/if a real small datacenter could born.
Are you horrified? Well, now try to see the recent history, when Amazon choose not to buy expensive and reliable SUN servers choosing cheap and crappy PCs or Google choose to save money with rackmounted velcro-assembled PCs with a built-in UPS, i.e. a naked velcro strapped one. See the history of buying cheap low quality ram because we have ECC, lo quality storage because we have checksums anyway and so on. Then think about some real recent datacenter fires and what they told about "how brilliant and safe" was their design.
Long story short: as we ditch big iron in the past, we ditch mainframes, it's about time to think ditching dataceter model as well for a spread set of small unreliable but many machine rooms at homelabs scale. This, like FLOSS gives us reliability, cheapness and ownership as well. This ALSO decouple hw and sw giants, witch is a very good things in terms of free markets. It will also force he OEMs to be flexible in their design and ditching slowly from modern crappy fats-tech also some giants deprecate [2] as unsustainable, for more balanced tech because well, such homelabs last much longer than typical ready-made PCs and craptops or NAS from the mass distribution market.
[1] buying from China directly instead of feeding local importers who skyrocket the price instead of lowering it when Chinese supplier lower them
- Getting a static IP and having your traffic not go through a NAT might be challenging or impossible depending on your home ISP. You can use dynamic DNS in some cases, but that's meh.
- If you want to send email from your server, you'll have a higher chance of it ending up in spam folders because of the residential IP.
- Home ISPs don't have SLAs. Your network was down for half a day without warning? Oh well. Things happen. We're sorry.
- Same goes for power, it's not as guaranteed at home as it is in a datacenter. Even if you install a UPS for your server, you can't be sure that the ISP's equipment will also work if you lose power.
Reliability, and depending on your work load the ability to use it commercially. Your house isn't going to have any meaningful reliability for either the network or power.
Your house will have one connection to the outside work, the DC should have multiple for redundancy (In case a digger goes through your network connection), and same for power any good DC will have their own redundant power supplies (Batteries, generators etc).
And if your server is serving a lot of traffic, your consumer home ISP might not be too happy about it. Theres a reason they split plans between home and business.
> Your house isn't going to have any meaningful reliability for either the network or power.
Not really a serious argument, but maybe worth looking at this these days. Mostly because I think as engineers we all have the desire to hit 100% uptime and reliability when in a lot of cases "close enough" might be a lot further than you think.
On the power end, a lot of people are getting to some form of local generation and storage (e.g., solar + batteries). Small backup generators aren't entirely unheard of. And on the network end, I'm probably not the only person that went to multiple connections when my ability to earn income became tied to my ability to be online from home.
As it sits right now, my power's not the most reliable in the world but because of that I already have a fair sized generator and a lot of fuel stored on-site (~5 days of gasoline, ~10 days of propane). I'm directly off of a fairly regionally important power line, so I'm usually fairly high up on the priority list during any major outage. (I mean, not "hospital" but if my power's out so are tens of thousands of others generally.) My grid power's rarely out for that long, and my local power even less since I have the generator. If I finally got around to adding some solar and storage to bridge between grid going down and the generator firing up I'd be able to pretty comfortable ignore power outages for over a week before I started needing to worry about making trips out to dump fuel in the generator or calling for some propane.
As far as connectivity, I've got three connections across two providers. All my traffic hits a real data center first so I can bond the connections (because residential) then flows to and from my house. One of the providers is wireless so the only single point of failure for cut lines is a few feet in front of my router.
I'm well aware this gets nowhere close to the reliability of a data center. I wouldn't run anything safety or life critical this way. I wouldn't run anything making a substantial amount of money this way. But there's a lot of stuff that fits in below "successful SaaS" and "life critical" that I think could tolerate an hour of downtime here or there that would get along just fine like this. And even if you think "well, you're clearly an insane person no one else is going to do that" I think there's still lots yet that could tolerate a day or two of downtime every year or two just fine. There are a lot of smaller services out there that businesses rely on that are hosted in AWS but aren't much more reliable than if they were running in someone's basement and they survive just fine.
I can't imagine it's actually illegal. Maybe cost-prohibitive and a bit noisy, but plenty of people have UPS systems for their local devices and backup generators.
I was gonna ask the same thing, but then she called me a usual broken people and I decided "not me" was a good enough answer :)
This was a tortured way to setup a colo. I bring rackable gear, no switch, and use the colo's "setup station" to do final config, then I rack the sucker. They can always VLAN me and my laptop together somewhere for a final check before I leave. I'd be worried if they couldn't.
> 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd.
Also, test that it's properly disabled with something like `ssh -v yourserver : 2>&1 | grep continue`, because there are a surprising number of ways for that to go wrong (did you know that sshd lets you Include multiple config files together in a way that can override the main one? I know that now.)
This. OVH's VPS had two .confs reenabling passwords. Now I know too.
default ubuntu 22.04 has at least one, too
I always remove all 'includes' in sshd config, just to be sure I will not get any surprises later...
I think doing the opposite is better usually, never making changes to the main /etc/ssh/sshd_config and always add your own customizations to /etc/ssh/sshd_config.d/; that way you have clear separation of your own and distro configs, which makes life easier especially when upgrading sshd. Although I'll readily admit that I don't myself follow that advise all the time.
This. And give your drop-in a prefix like 00-*.conf.
Fwiw I don't think SSH adds the include line upstream. Most distros add it now.
In situations where I have more than one box, one (usually a little embedded SoC) gets openbsd on it, and is used as a bastion host, for exactly the reasons outlined here.
The problems I hit with using Linux for this were different ten years ago, but, based on this thread, things got worse on that side of the fence.
and `sshd -T | grep -i password`
Get something like a PiKVM and drop all of the stuff about being very local since you can find a cheaper overall provider elsewhere and use smart hands once a year either for free or still cheaper than picking the local place you can drive to. Even if you do the things in this guide perfectly it'll break/hang/get misconfigured at some point and the PiKVM (or like) lets you remotely hard boot a box instantly without having to drive or open a ticket. It also enables you to easily reinstall the entire OS remotely if you need to.
If your server/device has an IPMI... get a PiKVM (or like) anyways. Not only will you last more than 2 seconds without being hacked but it'll have more functionality and be much faster.
If you're in the US there are lots of places in the Kansas City area that have ridiculously cheap pricing and it's decently centrally located in the country.
If you don't want to assemble a PiKVM yourself there's always Tiny Pilot: https://tinypilotkvm.com/
https://pikvm.org/
Thanks, I had never heard of PiKVM!
What do you mean by “smart hands?” Will they replace hardware that failed?
DC's staff, yes they can replace hw if you have spares or can mail/order one in.
She wrote a "part 2" just today https://rachelbythebay.com/w/2024/09/23/colo/
A bit less terrible way in my opinion:
Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes build their own). Bonus points if they offer KVM - as in remote console, not the Linux hypervisor. Also ask if they do hardware monitoring and proactively replace the failed parts. All of this is still way cheaper than cloud. Usually with unmetered networking.
Way less hassle. They'll even take your existing stuff and put it into the same rack with the rented hardware.
The difference from cloud, apart from the price, is mainly that they have a sales rep instead of an API. And getting a server may take a from few hours to a few days. But in the end you get the same SSH login details you would get from a cloud provider.
Or, if you really want to just collocate your boxes, the providers offer "remote hands" service, so you can have geo-redundancy or just choose a better deal instead of one that's physically close to your place.
This. I used to colo lots of stuff, but now mostly use Hetzner. But there are many in this space, and some of them even offer an API. And some of them (like Hetzner) also offer at least basic cloud services, so you can mix and match (which allows for even steeper cost cuts - instead of loading your dedicated hardware to 60% or whatever you're comfortable with to have headroom, you can load it higher and scale into their cloud offering to handle spikes).
The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)
One hurdle that many companies who have only known cloud hosting will face here is significant: how do you find a reliable, trustworthy datacenter? One who actually monitors the hardware and also has a real human if your networking access gets screwed or if you need a critical component swapped at 2 am on a Saturday.
I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone.
Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.
> how do you find a reliable, trustworthy datacenter?
drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself
That’s unfortunate. I consider SoC 2 compliance as a negative indicator of security (I’ve been on the vendor side of it, and have seen it have significant negative impacts on security and reliability in multiple organizations).
Ideally, there’d be locked doors, and the data center wouldn’t be subsidizing performative checkboxing.
I'm curious, can you tell us what the negative impacts you've seen are? Are there any audits that you can say are a positive indicator for security?
Spending all their security time/budget on box checking rather than actual security.
I'd rather see open ended red team pentest reports.
I think if you want to host in Europe then Hetzner is the clear choice. They won't monitor your hardware for you though, you need to let them know if something breaks and they'll replace it very quickly.
You're right you need to find a company you can trust.
And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).
I often recommend a "embrace, extend, extinguish" approach to AWS: Starting there for simplicity is fine, then "wrap" anything bandwidth intensive with caches elsewhere (every 1TB in egress from AWS will pay for a fleet of Hetzner instances with 5TB included, or one or more dedicated servers).
Gradually shift workloads, leaving anything requiring super-high durability last (optionally keeping S3, or competitors, as a backup storage option) as getting durability right is one of the more difficult things to get confidence in and most dangerous ones to get wrong.
Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.
> Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.
Is there an off the shelf implementation of this?
Minio used to support this, but removed the feature a while back. It was called “gateway mode”. Sadly, I know that doesn’t help much now…
https://blog.min.io/deprecation-of-the-minio-gateway/amp/
Every time I contemplate increasing my use of S3 or similar cloud services, I get annoyed at the extremely sorry state of anything on-prem that replicates to S3.
Heck, even confirming that one has uploaded one's files correctly is difficult. You can extract MD5 (!) signatures from most object storage systems, but only if you don't use whatever that particular system calls a multipart upload. You can often get CRC32 (gee thanks). With AWS, but not most competing systems, you can do a single-part upload and opt into "object integrity" and get a real hash in the inventory. You cannot ask for a hash after the fact.
I understand why computing a conventional cryptographically-secure has is challenging in a multipart upload (but that actually all that bad). But would it kill the providers to have first-class support for something like BLAKE3? BLAKE3 is a tree hash: one can separately hash multiple parts (with a priori known offsets, but that's fine for most APIs but maybe not Google's as is), assemble them into a whole in logarithmic time and memory, and end up with a hash that actually matches what b3sum would have output on the whole file. And one could even store some metadata and thereby allow downloading part of a large file and proving that one got the right data. (And AWS could even charge for that!)
But no, verifying the contents of one's cloud object storage bucket actually sucks, and it's very hard to be resistant to errors that occur at upload time.
I usually just use a suitable Nginx config unless there's a compelling reason. It means you "waste" the first read - you just let post/put etc. hit S3, and just case reads, but it's easier to get right. It's rare this matters (if your reads are rare enough relative to writes that avoiding the cost of the first read matters, odds are the savings from this are going to be questionable anyway - the big benefit here comes when reads dominate massively)
Well, S3 is hard to beat for our use case. We make a heavy use of their various tires, we store a somewhat large amount of data but only a minor part ever goes out.
The compute and network heavy stuff we do is still out of AWS.
That's pretty much the one situation where they're competitive, so sounds very reasonable. Some of their competitors (Cloudflare, Backblaze) might compete, but the biggest cost issue with S3 by far is the egress so if not much goes out it might still be best for you to stay there.
Sounds like (unlike most people who use AWS) you've done your homework. It's great to see. I've used AWS a lot, and will again, because it's often convenient, but so often I see people doing it uncritically without modeling their costs even as it skyrockets with scale.
S3 is a decent product with zero competition. You should keep s3, it’s a fair price.
S3 has plenty of competition. It can be a fair price if you rarely read and need it's extreme durability, but that leaves plenty of uses it's totally wrong for.
As a technical person working at datacenter and at still handle technical support requests at some capacity, interesting to read this stuff from the customer perspective. Good to know what is considered and important by customers. Maybe sales staff knows all that too well, but for me just invokes a smile and some pride in the job I do :)
The absolute trust killer is when your customer finds out about some issue on their own - even though you were supposed to monitor that. :-)
More important than raw uptime.
Back in the late 90s/early 00s when I was a precocious teenager, I ran a somewhat popular website. At some point it made sense to just buy a 1U rack moubtable server and having it colocated (commercial webhosting was expensive then). I couldn't find anyone to give me a ride to the datacenter, so I took a bus. By the time I got there my arms were numb from carrying the bloody thing.
There was a single security guard, I signed in and he gave me directions and a big keychain. The keys opened most of the rooms and most of the cages around the racks. To this day I remain mystified at the level of trust (or nonchalance) that security guard had in a spotty teenager.
Back in the early 2000s I had a job that required me to enter a certain phone companies' datacenter. But, I couldn't, because I was a consultant and it was an employees-only area. I had permission to make changes to machines in a rack but they didn't allow me to enter. There was a guard who would check badges but who didn't really check if he recognized people, and there were some people who had to enter and leave multipel times in a day. Myself, I couldn't get permission to enter via the correct channels.
With my director's unofficial approval I was allowed to _try_ to enter the datacenter. So I just walked very confidently towards the entrance, nodded to the security guard like all of the regulars who didn't bother showing their badges, and he let me in.
You can get in anywhere if you just carry a ladder.
https://www.youtube.com/watch?v=NiEMcjSQOzg
Having run hosting companies from the mid 90s as well, from memory this kind of thing was pretty normal, even in allegedly secure places like London Telehouse.
Quite a few of us in that era were juggling it with being students, so it wouldn't surprise me if the security staff were used to it and expected you to look young enough to be their kid!
Modern society is absolutely paranoid. Zero-trust policy, everyone is treated like a potential terrorist.
It has nothing really to do with that.
To this day I can get into pretty much any rack or room I feel like at datacenters everyone here has heard of. It just takes experience these days and a bit of charm. Plus having a million keys and staff rack combination codes doesn't hurt. These were freely given and simply added to my collection over time, nothing stolen or social engineered.
I've never done anything nefarious with these abilities, and no one I know has either. It's simply a matter of practicality when you staff a 150,000 square foot facility with 2 security guards who have no idea what they are doing.
If I (and many others) had wanted to, we could have caused multi-week/month outages you'd be reading about on the news with 5 minutes of effort. This is basically the status quo for any sensitive industry.
The world turns because 99.9999% of people want to give you a hug vs. hit you. Society falls when that ratio goes much lower.
That won't work everywhere. I've been to a datacenter in London around 2010 where the entry was similar to the automated airport passport booths. The doors would not open if you were not registered for a visit and there were no visible guards (I hope someone was around in case you got trapped...) I wanna say it was Telehouse West, but my memory is not great.
You're treated lile a terrorist when you are a law abiding citizen.
Actual criminal? Society will bend over to accommodate you.
They're absolutely correct:
" 1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd. If you just locked yourself out of sshd because you didn't install ssh keys first, STOP HERE. You are not ready for this. "
If you blindly followed the directions and got locked out, you can do exactly the same thing with other directions. You were not ready.
Related to this, is the advice I give everyone for editing the sudoers file:
First, open two superuser terminals. The second one is so if you fuck up the sudoers format so it doesn't parse and you accidentally 'exit' one too many times in the first terminal.
https://www.sudo.ws/docs/man/1.8.13/visudo.man/
"visudo edits the sudoers file in a safe fashion..."
The visudo command is often times available as well. It won’t let you save a malformed sudoers file. Finding other footguns though is an exercise for the reader.
It will however let you accidentally break "sudo su -" and happily tell you to go fuck yourself afterward.
sudo -s or sudo -i
That will not change sudo's attitude toward you once you've bungled the sudoers file.
I would add: disable root logins, create exactly one account that can login, require a password for sudo
At least it doesn't say to set PermitRootLogon and remove the root password :)
AKA, "We are compatible with passwordless authentication mechanism." Or SSO, depending of the use case....
Le sight.
And I find this to be mean-spirited ("snarky") and the opposite of instructive. Perhaps I am new, or rusty, and I do not know about SSH keys or remember them. Bang! Trapped! And then how is a newbie to trust the rest of this, much less learn?
Good instruction will tell you why and the consequences of missing a step. Perhaps some options.
But perhaps I want too much, the title does include the word "terrible," after all.
If you are new or rusty, Stop. You are not ready for this.
Seriously. Not knowing or remembering abut ssh keys is way below the minimum baseline of skills that are needed to manage a server connected to the internet.
Managing colo-ed servers when you're only just experienced enough to know about and know how to set up ssh keys without being told _is_ a terrible idea.
This is wrong. I may not be "ready" for it, but I need it anyway. What then?
What is the reason NOT to educate me about SSH keys instead of just telling me to go home?
Everyone has to start somewhere.
> Everyone has to start somewhere.
(This was my path. This may not be yours.)
If you want to start somewhere, start from the machine you already have: Your PC.
Work up to using a VPS. A DigitalOcean droplet or a Hetzner VPS, or any other provider.
Learn how to generate public-private keypairs. PuTTY comes with a tool to do this (puttygen.exe). Otherwise, learn how to use ssh-keygen.
Learn how to move your private key securely to the VPS. Learn how to disable password logins to your VPS, & moving the SSH port to anything > 10000.
Do this for multiple VPSes.
Sorry, but you're still well into the "don't know what you don't know" stage of learning, which is _not_ the time to become responsible for a colocated server.
Go home, from the colocation idea.
Start with way less expensive VMs from your choice of hyperscaler or el-cheapo-vms.com, or in a homelab with old hardware or RasPi/IntelNUC type devices.
If you "need" colocation, you need someone who knows what they're doing to set it up and maintain it properly.
Or just go ahead and wing it. Maybe I'm wrong. Maybe Rachel's wrong. Maybe you're "The One".
Incorrect. Wildly incorrect. I used to co-lo servers, it has been a long time. SSH Keys were not in vogue back then. I did use them at one point and, because I haven't done it in a while, it slipped my mind. I'm just rusty.
There's just nothing wrong with adding a few steps to do your keygen, upload your public key, save your private key on a thumb drive or whatever, and so on. Please, tell me why that should NOT be added to a tutorial. Whom does it harm? What is the downside?
I can't see anything but Crappy Gatekeeping, but please, come up with a decent reason not to add just a few more instructions.
(Although now, apparently, keys are out for some folksand certs are coming in vogue. Who knew?)
How many _other_ important things have "slipped your mind"?
This isn't about ssh keys specifically, it's using ssh keys as a canary - an example of a thing someone who has internalised enough Linux sysadmin and security knowledge will automatically get right. It's shorthand for all the other hundreds of things that need doing to properly secure a public internet connected Linux machine.
This is a way bigger thing that adding a few lines explaining ssh keys.
Perhaps firewall config has also "slipped you mind"? Or TLS certs and renewals? Or shutting down unneeded service started by your distro? Or managing logfiles? Or monitoring disk space/memory usage/cpu usage? Or keeping your JVM/Nodejs/Python/PHP/whatever up to date with security patches? Or maintaining your software BOM and all the security update notification channels for everything you're running whether you installed it ir whether it got bundled in as a dependancy of something else you installed? (Think zx or Log4J)
Or maybe's since you're just "rusty", you're looking for all the sysvinit files you remember being important, not realising your chosen distro now uses systemd. Or your previous experience was on machines from before the era of speculative execution attacks, or from when it was considered acceptable to hash passwords using crypt or MD5?
I don't know when you were doing colo servers while ssh keys "were not in vogue". In the late 90s I was flying annual round the world trips to visit our colo facilities in London, New York, and San Francisco - with half a dozen hard drives in my luggage because replacing raid1 arrays was a better/cheaper solution for us than uploading over our available bandwidth over 56k modems or ADSL. All those machines had ssh keys. It was standard practice.
But yeah, maybe you're right and my 30 years experience and hard earned advice is just "gatekeeping". Same with Rachel the article author's decades of experience (at way higher levels than mine). I'm sure that your assertions of it being "Incorrect. Wildly incorrect" is not another one of the things that've slipped your mind or that you're a bit rusty on. Feel free to assume that having ssh keys "slip you mind" doesn't mean you have a lot to learn (or relearn) before being capable of securely/professionally managing a colo-ed server.
Say you get stuck at this point, and can’t fix it yourself.
That’s a strong indicator you won’t be able to support the setup moving forward. What if apt wedges some firewall rule, or the machine starts OOMing?
Or all your cat pictures get crypto lockered.
Or your user database full of PII gets popped.
Or your html/css/javascript gets modified to send all your supposedly secure Stripe checkout credit card details to eViLHackerz.ru
(Or your unmetered 1GBit network connection gets used to host p0rn or stage attacks for Mossad on hospitals and water utilities and power stations...)
Then your software will be set back by a day! The horror!
I'm writing a how-to for colocating for people who're closer to newbies. In this case, Rachel's article did suggest it's not for newbies.
The great thing about having unfettered physical access to hardware is that you can easily recover from mistakes like this. No need to rebuild that EC2 instance. No need to beg a hosting company for IPMI access. You can just pull the plug and try again as if were your own PC.
On the other hand, it's nice to be able to take a no-reboot AMI of the machine you're futzing with, risk breaking it, then just re provision from that AMI if you fucked up. And if you're chasing three or four nines of uptime you do the same but provision a new instance and futz with that one while your working one keeps on keeping on. Even big instances are cheap enough by the hour that you can afford to run and extra one while testing potentially breaking changes.
Isn't it the same with something like idrac?
Does anyone have a sense of how much colocation costs? Assuming you have a modest setup that would otherwise run on a home server or VPS?
The most difficult step I find is just barely mentioned, finding colocation space at reasonable price is difficult these days.
I faced the same problem, until I found a non-profit community-operated housing provider (a registered computer club/society that has been operating since at least the early 2000 who is also a LIR (https://en.wikipedia.org/wiki/Regional_Internet_registry#Loc...), and they give shelter, bandwidth and IP address space to anyone who participates in covering their operational costs.
I gladly do, and it's the best hosting experience I've had so far, and I used to have rented dedicated iron from Server4You, Hetzner, Webtropia and the like from 2005 on. Maybe there's a similar hidden gem in the area you live in, and you just do not know about it? :) Mine flew under my radar for nearly 20 years, and even though I knew they existed, I was not aware they'd colo other peoples' boxen at very fair rates.
How did you find your local non-profit? Is there a resource you can share?
> finding colocation space at reasonable price is difficult these days.
The author of the post seems to be living in the bay area. It's easy when the number of nerds per km² is high, disposable income is even higher, and driving a car for 2 hours is considered "next door".
I think for losers like myself, living in the middle of nowhere with low disposable income, the best solution is just to rent out a dedicated Kimsufi box (from OVH), or a server off the Heztner auctions. (Or whatever is the equivalent in North America) It's much much more cost effective than collocating.
> the bare minimum required to run your own hardware in a colocation environment
i remember the look in the admin's eyes when they asked "alright, what kind of hardware are you looking to install?" and I said "oh i have it right here" and pulled two Intel NUCs out of my backpack
> Consider bringing a screwdriver and a flashlight (any halfway decent place will provide those for you, but you never know).
two multitools minimum, sometimes you need to hold a nut with one while loosening the bolt with the other
the best space is the one that is right next to a Frys/Microcenter/whathaveyou
> "oh i have it right here" and pulled two Intel NUCs out of my backpack
NUCs would be like nirvana after some of the jerry-rigged crap I've seen dragged into facilities. Maybe I'd like them to have a secondary power supply, but then again you had two, but they'd make a fantastic little router for something not too traffic heavy. Lots of utility use cases for something like that in a proper facility.
> I'd like them to have a secondary power supply, but then again you had two
yup! it was cheaper to bring two NUCs than try to get N+1 power redundancy set up for "atypical" (not u-rack) systems :)
then they started making 1U NUC rack mounts...
Don't forget hearing protection! All the fans in there make it crazy loud. Facilities often supply earplugs, but consider using it with over-the-ear protection.
Interesting article.
> Plug everything in and turn it on.
Most server racks will have C13 or C19 outlets for power distribution. For consumer-type devices designed to plug into a wall socket, suitable cables or an adapter strip would probably be required.
IPMI/BMC is a must if plan to keep the server running longer than a release cycle of your Linux distribution.
Also remember ask your provider to assign you a private vlan and vpn to it for your own access only.
A working BMC is great, but I sure wouldn't want it facing the open web. The colo would either need some sort of private network, or you'd need to add a router and firewall of some sort to lock down access to the BMC.
Missing step -1: find someone to get on your staff that knows what they are doing, or at least find a contractor with a good history of doing this kind of thing. After a certain point, you really shouldn't wing it. But if you're at that point, you should know if you actually need your own hardware, or if public cloud or other options are good for you. Some need their own, most don't. There's a lot of nuance there that is difficult to express in a short HN comment.
Agreed. It blows my mind a bit that there is a generation of folks that only know Cloud vs Colo. Colo is a walk in the park compared to cloud, assuming you've done basic PC building and networking. Heck, Cloud can be a walk in the park too for most use cases. If you don't have time to learn it yourself over a month or two, there should be folks available to consult. A lot can be brainstormed in an hour.
Remote power management can be a godsend. If you can get an ipmi console in, you want it.
Ipmi is a bit of a double-edged sword. Network-connected access to a serial console (including uefi/bios console redirection) and the reset button can be a total lifesaver, I agree. I wouldn't want to be without a serial console and remote reset button either.
But ipmi cards are little network-attached linux boxes which run prehistoric kernels and userspace, exposing as many services as the half-wits who put together the firmware image can shovel in, and are rarely if ever patched by the vendor unless there's some really public scandal.
The standard thing to do is to isolate them on some kind of private management network in an attempt to shield the wider internet from the full majesty of the firmware engineers' dazzling skills, but that might be harder to do in the simple 'beginner' scenario Rachel describes.
One good simple version when you get up to two servers instead of just one is to cross-connect so each machine has ipmi access to the other, but neither ipmi service is exposed to the wider world.
Even if your server does not have proper IPMI, most colo providers have an assortment of ip-kvms, you can ask for those and access your server from any modern browser.
I'm pretty suspicious of trusting the firmware on things like that either, to be honest. (Personally, I tend to use serial cables and a little custom board to turn serial breaks into resets, but I know my NIH instinct is probably a little on the high side. It would be unreasonable for me to suggest someone new to colo start making random extra boards to stick inside their machines!)
You do not need to stick it inside, it plugs into monitor and keyboard ports. Power-on and reset are usually done manually on request. And you do not need it plugged in all the time, only to reconfigure network or boot values.
That's definitely a level of remote-hands intervention I've not had from the handful of places I have racks of colo kit. You're right, it might be something some of them do for other customers though; I've never really asked. They'll certainly push a reset button, swap a drive or rack a server that's shipped to them. (Though hardware's reliable enough nowadays and I have go to data centres so rarely that it's a bit of a fun novelty, and I like doing it myself.)
I've been colo'ing forever. I'd consider a Lantronix spider ipkvm available on request to be like minimum viable colo service. It's usually up in 5 minutes when I submit a ticket to the NOC.
I bet your provider has something like that. It's a godsend when you screw up, say, the hypervisor running state somehow and need bare metal access to unbork it :)
My version of this is to use a serial console server for direct access, rather than vga + keyboard. If the kernel is still running fine but network access is down, I have a getty running on the serial port. If the kernel has locked up too, I hit the reset button by sending a 500ms+ serial break (special circuit, highly recommended) and the bios has serial redirect, so I can do stuff like hit del to drop into bios setup or uefi shell at that point.
But I expect you're probably right some or all of the providers we use do have something like that, as I speculated in the previous post. I've just never understood the point of vga-type stuff when bios/uefi serial redirect exists and serial console is more convenient anyway once the kernel has started, so never asked the question.
Yeah, I like the idea of a bmc but the implementation sort of sucks. What I want: a small soc pony engine that controls the main computer. On this soc I want to install my own os. I like openbsd so that would be my ideal os, but the point is it should be my os not the vendors.
It is indeed very helpful when it's needed - but most IPMI is horrifically insecure, so usually it's not connected directly to the public internet. Instead it's on a separate network with special VPN access.
As this document focuses on keeping things simple, they don't have the isolated network / VPN needed to use IPMI.
Remote controlled power strip. Turn off and on a server that looks bricked to see if you can recover without remote hands.
What's the importance of having a switch/hub? Is it because you may want to add more servers later, but the colo host only provides one port in their router?
So you can plug your laptop into it at the colo when you set it up so you can verify that the server is working right and can get to the internet while you are still there. And if it isn't working right, it'll be a lot easier to keep everything plugged into the switch, rather than swapping between your laptop and the colo's network every time you try something to fix it.
But yes, if you think you might want to add other machines later, preemptively putting a switch in there will make it tons easier.
Basically. Need to pay for a bigger pipe. Same applies to power as well, gotta pay to get more capacity.
Also if you want to manage the IP ranges provided amongst >1 server.
I'm writing up a how-to about colocating using one of those Pi colocation offers. Glad to see colocation get some more attention!
My tip is adding hot spare disks so you don't have to replace broken ones.
For example, in ZFS you can mirror disks 1 and 2, while having 3 and 4 as hot spares with:
How about make it so you can smash the whole rack with a sledgehammer or datacenter burn down and system hums along like nothing happened.
No; because that costs at least an order of magnitude more money than a simple colo setup.
It is also useful to configure mutiple IPs on the same interface, one for Colo, and one for your lab. You can do this on all OSes, even on windows.
And if you failed to do this and all advices in the article: then kindly ask colo provider to attach ip-KVM (if your "server" is not actual server with ipmi/bmc), 99.9% of places have these. And most of them will provide it free of charge for limited amount of time.
Documenting and testing the hard reset/reboot procedure, as well as any expectations, for your colocated gear sounds like a good thing to add to this list.
Test to make sure it'll actually come back after a hard reset.
Don't do what I did and put a Cisco router on a remote mountaintop radio site with the config register still set to 0x2142, and then go home.
Also, check what happens when mains power disappears and returns.
Systems Administration the Hard Way... it's an uphill hike, both ways, and in the snow :)
Hey, that's just an excuse to go for another hike! :)
3 hour drive in warm weather, or an expensive helicopter ride in winter!
Is colo necessary of you have a 1000Mbit symmetrical provider at home?
I selfhost a bunch of personal stuff at home and have 1500M/940M at home, and I don’t think bandwidth would ever be the issue.
Frequent IP changes and the residential IP block could make mail hard.
If I was looking to host something that mattered, and was a little more public, I would move that more for my own sanity. Not being able to find me via IP, network uptime guarantees, more robust backup power, etc.
If you were some cloud admin and put your website on a resume or something, and it was down, I might use that to jump to conclusions.
Is this cloud repatriation?
Imho, the easiest way is to get a Mikrotik switch, set up Wireguard on it, allow only Wireguard in, then use the Wireguard Mac App. Build your machines with AsRock Rack or Gigabyte motherboards. If you have existing, then add a PiKVM. You'll get an IPv4 with a half-cab in most DCs.
HE will give you a full cabinet for $600/month https://he.net/colocation.html?a=707336661752&n=g&m=e&k=hurr...
I suggest a very different reasoning: we know WFH works, we also agree that in large part of the world we have enough sunlight for p.v., battery storage could be cheap [1] and FTTH connections start be be very good for modern time. Who have already understood where I want to go?
Well, we could run decentralized systems at home for SMEs/Startup common needs from modern homes of remote workers, just renting them a room with proper local equipment the modern remote worker probably already have (VMC, A/C, "the big UPS" from domestic p.v.) with not much more expenses than using someone else computer also named "the cloud". Then if/when needs mount some sheds with p.v. + local storage and good connections can pops up. Who need a CDN in such setup? Then/if a real small datacenter could born.
Are you horrified? Well, now try to see the recent history, when Amazon choose not to buy expensive and reliable SUN servers choosing cheap and crappy PCs or Google choose to save money with rackmounted velcro-assembled PCs with a built-in UPS, i.e. a naked velcro strapped one. See the history of buying cheap low quality ram because we have ECC, lo quality storage because we have checksums anyway and so on. Then think about some real recent datacenter fires and what they told about "how brilliant and safe" was their design.
Long story short: as we ditch big iron in the past, we ditch mainframes, it's about time to think ditching dataceter model as well for a spread set of small unreliable but many machine rooms at homelabs scale. This, like FLOSS gives us reliability, cheapness and ownership as well. This ALSO decouple hw and sw giants, witch is a very good things in terms of free markets. It will also force he OEMs to be flexible in their design and ditching slowly from modern crappy fats-tech also some giants deprecate [2] as unsustainable, for more balanced tech because well, such homelabs last much longer than typical ready-made PCs and craptops or NAS from the mass distribution market.
[1] buying from China directly instead of feeding local importers who skyrocket the price instead of lowering it when Chinese supplier lower them
[2] https://unctad.org/system/files/official-document/der2024_en... of course suggesting the wrong answer
Why would you do this instead of having the hardware at your house? For better/faster connection?
- Getting a static IP and having your traffic not go through a NAT might be challenging or impossible depending on your home ISP. You can use dynamic DNS in some cases, but that's meh.
- If you want to send email from your server, you'll have a higher chance of it ending up in spam folders because of the residential IP.
- Home ISPs don't have SLAs. Your network was down for half a day without warning? Oh well. Things happen. We're sorry.
- Same goes for power, it's not as guaranteed at home as it is in a datacenter. Even if you install a UPS for your server, you can't be sure that the ISP's equipment will also work if you lose power.
Reliability, and depending on your work load the ability to use it commercially. Your house isn't going to have any meaningful reliability for either the network or power.
Your house will have one connection to the outside work, the DC should have multiple for redundancy (In case a digger goes through your network connection), and same for power any good DC will have their own redundant power supplies (Batteries, generators etc).
And if your server is serving a lot of traffic, your consumer home ISP might not be too happy about it. Theres a reason they split plans between home and business.
> Your house isn't going to have any meaningful reliability for either the network or power.
Not really a serious argument, but maybe worth looking at this these days. Mostly because I think as engineers we all have the desire to hit 100% uptime and reliability when in a lot of cases "close enough" might be a lot further than you think.
On the power end, a lot of people are getting to some form of local generation and storage (e.g., solar + batteries). Small backup generators aren't entirely unheard of. And on the network end, I'm probably not the only person that went to multiple connections when my ability to earn income became tied to my ability to be online from home.
As it sits right now, my power's not the most reliable in the world but because of that I already have a fair sized generator and a lot of fuel stored on-site (~5 days of gasoline, ~10 days of propane). I'm directly off of a fairly regionally important power line, so I'm usually fairly high up on the priority list during any major outage. (I mean, not "hospital" but if my power's out so are tens of thousands of others generally.) My grid power's rarely out for that long, and my local power even less since I have the generator. If I finally got around to adding some solar and storage to bridge between grid going down and the generator firing up I'd be able to pretty comfortable ignore power outages for over a week before I started needing to worry about making trips out to dump fuel in the generator or calling for some propane.
As far as connectivity, I've got three connections across two providers. All my traffic hits a real data center first so I can bond the connections (because residential) then flows to and from my house. One of the providers is wireless so the only single point of failure for cut lines is a few feet in front of my router.
I'm well aware this gets nowhere close to the reliability of a data center. I wouldn't run anything safety or life critical this way. I wouldn't run anything making a substantial amount of money this way. But there's a lot of stuff that fits in below "successful SaaS" and "life critical" that I think could tolerate an hour of downtime here or there that would get along just fine like this. And even if you think "well, you're clearly an insane person no one else is going to do that" I think there's still lots yet that could tolerate a day or two of downtime every year or two just fine. There are a lot of smaller services out there that businesses rely on that are hosted in AWS but aren't much more reliable than if they were running in someone's basement and they survive just fine.
Typically having the necessary batteries and generators at your house is illegal.
I can't imagine it's actually illegal. Maybe cost-prohibitive and a bit noisy, but plenty of people have UPS systems for their local devices and backup generators.
That's just nonsense and is not based on fact at all.
What audience was this written for?
I was gonna ask the same thing, but then she called me a usual broken people and I decided "not me" was a good enough answer :)
This was a tortured way to setup a colo. I bring rackable gear, no switch, and use the colo's "setup station" to do final config, then I rack the sucker. They can always VLAN me and my laptop together somewhere for a final check before I leave. I'd be worried if they couldn't.
[dead]
Forget the switch and pi and get a ubiquity router. Much more powerful and simple to setup. Does require some knowhow.
Also you could see if your local hackers space has a server rack