Current Inbox status jeremy has 19 items in his inbox

Posted by jeremy, at 2015-08-09 08:05:16

back to real DHCP and DNS services
I've had a network in my house for nearly twenty years now. I initially had a linux server which functioned as internal DNS and DHCP, among other things. As I virtualized functionality into VMs for services such as SMTP and HTTP, I kept my single DHCP/DNS server, which also functioned as my firewall.

The drawback was that when the main server was rebooting or down for maintenance, the whole network was out of action. I eventually moved to a Soekris box running m0n0wall. As that got long in the tooth, I picked up a PCEngines Alix box (2d13), and put pfsense on it.

This worked very well for years -- right up until Friday afternoon. We'd had a power outage, so the house could get 200A service, and afterwards everything came right back up. The Alix box booted up, started servicing DNS and DHCP requests, and then...stopped. No error messages, no strobing LEDs on the box, nothing.

It took me a couple of hours of trying backup CFcards (which contained working pfsense and m0n0wall images) to pronounce the culprit to be bad hardware. Thankfully, I had a wireless router on hand. But while the wireless router could do DHCP, it could only forward DNS requests to the outside. Further, the pfsense box contained all my MAC-based DHCP host definitions.

Microcenter doesn't carry PCEngines equipment, so I had to think over my solution from the past. Why had I moved from a single point of failure to a new single point of failure? Because of the promise of better stability. It WAS more stable, right up until it bellied up. How do real ISPs handle this? Redundancy. This led me to my actual solution: I'd just have DNS and DHCP servers in failover inside my network.

This was less of a push than I'd feared. I still had zone files from a few years ago, which had a bunch of correct data in them, and I had some MAC mappings in the old /etc/dhcp/dhcpd.conf files on my primary server. I also had a primary (fileserver, virtualization) server, and a secondary (fileserver, virtualization, minetest) server still on hand, and still running all the time.

So I dusted off those config files, googled failover on DHCP, and googled slave replication for DNS, and rejiggered the config files. Then I threw them into a puppet module, put that into git, and adjusted my r10k configuration to grab the two new modules, and enforce the service on my two servers.

I activated DNS first, made sure it was working, then had to get DHCP going. I didn't REALLY want a DHCP war on my network, but the addresses would all be on the same network, so it really wouldn't matter. Still, I fired up the failover DHCP peer first, and got a mild surprise: it was so much faster than the WLAN router that it started answering DHCPREQ packets before the WLAN box could. And, thankfully, they worked. Then it was just a matter of kicking the puppet agent on the primary box, confirming that DHCP was working there, and deactivating DHCP on the WLAN router.

So, now I'm back to where I was five years ago, but in a redundant configuration, and I can more easily use any commodity piece-of-junk as a gateway host, as long as it supports port-forwarding and a static IP. It feels good to have my config back as text files, instead of junk thrown into a gui with click-and-drool mouse gestures.

Posted by jeremy, at 2015-02-04 07:52:22

no special snowflake uids here (anymore)
so, amongst my other sysadmin duties are a number of hand-build ubuntu workstations. Some my advice on using puppet to manage userids:

Do not use any userid under 1050, nor any groupid lower than 1050.


puppet has a blast shuffling uids and gids that already belong to users or groups -- you generally get some chicken-and-egg issues going on. Nevermind persistent machines where users have processes running through screen (this may be a good time to apply those kernel patches you\'ve been putting off, and reboot the box...) Set your admins with a nice high number in the 8 or 9000 range, somewhere LDAP or AD will not conflict.

I just rolled out a change across a whole bunch of lovingly handcrafted workstations, wherein many different users where uid 1000. Lovely. That will not be a problem moving forward, but i still touched more machines than i would have liked. I could have saved myself a lot of time if i had not tried to keep one particular account (mine, i must admit) at uid 1000, while forcing everyone else that shared that uid to a new one. I should have told myself that i do not get to be a special snowflake either! Had I admitted to myself that i would have to be running a find command to change ownership on MY files as well as all those other pretenders to the throne:

find / -uid 1000 -exec chown janders {}\;

i'd have gotten it done in almost one pass. meh.

Posted by jeremy, at 2012-09-05 10:31:47

data recovery
So, for quick-n-dirty data recovery off of a failed drive, try combining ddrescue and foremost. (assuming failed drive is at /dev/sdi, and the partition you want is /dev/sdi1)

ddrescue --no-split /dev/sdi1 ./failing_image mylogfile

then,

ddrescue --direct --max-retries=3 /dev/sdi1 ./failing_image mylogfile

THEN,

foremost -v ./failing_image

Browse through your "output" dir to see the stuff that got saved.

Posted by jeremy, at 2012-08-15 11:06:13

I'm tired of hearing about the 'nanny state'
Don't like the Nanny State?

Our nanny state provides us with socialized fire and police protection. Indeed, socialized fire protection is barely a hundred years old. Would it be better to return to the days of private fire insurance, and fire companies who won't lift a finger to help until the home of a paying customer is ablaze? As a nation, we've decided that the best way to handle this is centrally, with tax dollars -- even though that means spending money on things that some citizens would oppose, like rescuing a swimmer who defied a "no swimming" sign, or fetching a cat out of a tree, or cutting a morbidly obese person out of their home.

Without the army of lawyers and protections provided by patent law -- another example of our "nanny" state -- how much innovation do you think would occur? Any independent inventor would get about 3 months to manufacture and sell their product before a bigger business undercut them on the exact same item.

Would you trust medicines without an FDA to assure at least a modicum of safety in their production? Considering that baby formula used to have arsenic in it, I'm not sure I would. I'm grateful every day for the purity of the medications available to us in this great country, and that is thanks in no small part to the efforts of the FDA.

And don't think for a second that the Department of Defense isn't a part of the Nanny state. After all, doesn't a good nanny keep their charges from getting beat up by the bully down the street? Isn't that what the Navy is doing when it suppresses piracy, for example?

"Nanny State" is as useless and loaded a term as "welfare queen." Perhaps tellingly, Welfare Queens never actually existed either.

Posted by jeremy, at 2012-07-26 08:48:37

wrote a helmet review
I just bought a Cyber Helmets US-97 F-bomb helmet, and wrote a review of it. Enjoy!

Posted by jeremy, at 2012-06-04 13:01:23

Silly LibreOffice
So, when I installed LibreOffice 3.5, after uninstalling Libreoffice 3.4, it left the user-settings folder. This prevented it from starting -- it generated some complaint about MSCVRT.dll. I had to manually delete C:\Users\MYNAME\AppData\Roaming\LibreOffice in order to get LO 3.5.4 to successfully start.

Posted by jeremy, at 2011-10-06 16:31:12

yes, it can be done
BigFix (now Tivoli Endpoint Manager) does not officially support any way of completely removing settings from the configuration file.

So I set about making my own way. Using sed and shell, this script will eliminate settings from your besclient.config file. It needs files in /tmp to be executable, it needs sed to exist, and your configuration file should be in /var/opt/BESClient/besclient.config.

While this is not hyper-tested, I have been using it for a while in my own lab. Enjoy, and give credit where due.

Posted by jeremy, at 2011-05-05 12:34:19

Marvin Windows customer service rocks.
Three years ago, we had a bay added into our kitchen. Valet Building Services out of Rogers/Hamel did the work. When they finished it, they were demonstrating how to remove one of the sashes from a window, and damaged the track. They replaced a part then, but the sash would always sag when lifted to the top. I called repeatedly, but the owner had closed down the company, and fixing it was apparently above and beyond.
Hamel Lumberyard, however, was happy to send a guy out to look at the windows. They filed a repair request with Marvin, who sent a guy out today.

The Marvin guy showed me how the carpenter from Valet had used brads which were too long, and damaged the track. He'd also driven brads through some of the plastic weather-resistant covering on the sill.
Marvin would've been totally within rights to refuse the repair, but they STILL covered it under warranty -- even though it was carpenter error. I'm glad I went with Marvin windows for the whole house! I'm also glad I had Aspen Building Contractors do all the other windows for me. The carpenters at Aspen know how to do things correctly.

Posted by jeremy, at 2010-11-24 10:17:41

another manual I might need to find
The cd player in the kids' room is an Insignia A-1112. Long ago, I lost or recycled the manual. Today, I had to join some hokey site to get one, so I figured I'd put it up right here: Insignia A-1112 Executive CD Player PDF Manual, so I wouldn't have to go looking for it again.

Posted by jeremy, at 2010-09-16 10:27:35

bluetooth on win7_64
I've had a bluetooth dongle in my PC for years, but when I upgraded to win7 (directx 10, you know), I didn't bother getting everything configured. After looking at the bluetooth headset on my desk for the umpteenth time, I decided it was time to either configure the SBH500 with my PC or sell it. So I configured it -- took me a while, but the trick was to update the bluetooth software on win7, using new broadcom drivers: http://www.broadcom.com/support/bluetooth/update.php -- make sure you actually HAVE a broadcom chipset bluetooth dongle. After that, the headset magically started work. Oh, and to put the SBH500 in discovery mode, turn it off, hold down the button with the image of the headset, and turn it back on. Hold the button down until you get a steady blue light.

Posted by jeremy, at 2010-06-21 20:50:46

silly digital character watches
Years ago, one of my kids got an Armitron Scooby-Do watch. The instructions, however, have been lost in the sands of time. Googling for said instructions is futile. A little common sense, however, and you'll find this link: http://www.armitron.com/docs/ArmitronLadiesDigital.pdf -- the instructions for the Ladies Digital watch -- which is what the Scooby-Do watch uses as guts. Hopefully, this link will save you the googling I have to do twice a year, when the clock changes. Silly daylight savings time. I have also cached the manual right here, incase Armitron pulls their manual. Please use their site first, though.

Posted by jeremy, at 2010-04-15 21:07:13

HP SIM, PSP v8.25 and my memory
I spent a few weeks testing and trying to implement full integration with HP SIM (Systems Insight Manager) and a fleet of about 650 RHEL4 and RHEL5 boxes (we didn't even try with the RHEL3 machines) at a major retailer.
Through this time, HP support first said that PSP v8.15 would work. Then, when that didn't work, we upgraded first to v8.20, then v8.25. Again, unsuccessful. We DID manage to get full SIM integration by installing the v8.30 hpsmh and smh-templates RPMs on v8.25 machines, but that's a bit of a hack.
v8.30 was the first PSP that we installed that got us proper SIM integration -- full reporting back to the SIM server and automatic dial-home capability, and integrated access to the iLO console.
(while I did this work nearly five months ago, I had to remind myself of it today, so it goes here, so I don't forget. I've yet to lose my blog, after all)

Posted by jeremy, at 2010-04-15 21:02:01

aix thinks you entered your password wrong, really.
Had a situation where a user was correctly entering their password to login to an AIX 5.3 (oslevel 5300-09 is no longer affected by this bug) box over openssh, yet was getting tagged with an unsuccessful login. The user would enter their password once, but an unsuccessful login was being declared, followed 3-4 seconds later by a successful login -- even though they didn't get reprompted.
After _much_ analysis of machines, I found the issue: On those servers that would double-attempt the login (once unsuccessfully, once successfully), someone had added the line:

UsePrivilegeSeparation no

Once I disabled that line, by prepending a # character, and reloaded the config:

kill -HUP `ps waux | grep sshd$ | grep -v grep | awk '{print $4}'`

(that command is done from memory, by the way, and is special to AIX), everything worked just fine. This goes here because I didn't find a single tip on this solution ANYWHERE that google could find.

Posted by jeremy, at 2010-01-18 11:19:25

New stereo in the car
I finally got around to putting the new stereo in my 2002 Mazda Protege5. I also took pictures and explained what I did. Hopefully, this page can benefit anyone else who is looking to do the same thing.

Posted by jeremy, at 2009-12-04 07:59:57

vmware and read only filesystems
File this under "weird":
Years ago, due to some timing issues in various SCSI drivers, Linux guests running under VMware would decide that they were having hardware issues, and force their drives (/dev/sda, /dev/sdb) into read-only mode. This sort of thing makes grand sense when you're dealing with real hardware. On vmware, though, we might as well trust that the device is good. This also happened on Xen and VirtualBox, too. Basically, if the Host OS didn't respond fast enough, the guest decided that there had been a hardware failure.
RedHat fixed the issue, SuSE fixed the issue, all was well. This was back in the RHEL 4.6/5.1 era.
Except...the bug is back. I'm running into it on fully patched RHEL 4.8 and RHEL 5.4 machines. Careful digging through mailing lists suggested that adding "barrier=0" into the options for each filesystem, in /etc/fstab would fix things. Sure enough, it did. I'm happy to report that HP tech support has informed me that they're now using my solution at OTHER customers, with good success. Just goes to show -- just because a bug is fixed, doesn't mean it STAYS fixed.