Windows, Microsoft, etc.
I’ve been interstate since late last week (hence no activity on the blog), doing an upgrade for our major customer. While on site, I was bitten (hard) by three Windows problems…
1) XP provides no useful feedback when joining an NT domain… When the machine joins successfully, it displays “Welcome to the domain domain.” (we call our domains “domain”). When it fails, it displays exactly the same message. Whether the join worked or did not, there is always an error message in the event log indicating that it didn’t work. The UI always suggests it works, the Event Viewer always suggests that it fails… Only way to know one way or the other is to reboot and see if you can log in. Whether this is caused in part by the second problem below or not I don’t know, but either way its stoopid.
2) User and Computer account maintenance always happens on the primary domain controller (PDC), logins (both user and computer logins) always happen on the backup domain controllers (BDC). PDC and BDC’s are synched periodically to ensure consistent user data on all domain controllers. Problem is that when leaving/joining the domain, modern PC’s reboot waaaaay too quickly for the synch to happen. Which when combined with the problem above makes it fscking near impossible to a) determine whether the domain join failed and that’s why you can’t log on, or b) the join worked, but the synch hasn’t happened, and that’s why you can’t log on…. “net accounts /synch” is your friend, but when server and workstations are at different ends of the building its a pain. Brute forced it in the end by shutting down the BDC (which wasn’t being used at the time), so all changes and logins happened on one machine, the PDC.
3) Previously on all of the NT desktops, we had mapped drive P: to be the drive we ran all our POS software on… We had desktop shortcuts to P:\POS\POSLoader.exe on all of our machines. The new server I was installing was to run our POS software and databases, so I wanted to map drive P: to the new server, but I wanted the stuff on the old P: drive to still be accessible in case they ever needed it. So I did what anyone would do, and I mapped O: to the share on the old server and P: to the new share on the new server… However I’d assumed that Windows would leave the desktop shortcuts the hell alone, after all they clearly used DOS paths, not UNC paths… WRONG! When Windows saw the new O: drive it changed the desktop shortcuts to O:… This caused many dramas when everyone came in to start working Monday morning, and I discovered they were doing so on the old server not the new one… Fortunately I discovered this literally 5 minutes before I was going to delete the SQL Server databases on the old server, otherwise all hell would have broken loose.
Hope that saves someone some pain… At the very least I had to vent 
A tester working for Microsot picked up on Scotts rant (which I linked and responded to in my last entry. If I was trying to explain WinRot to a tester who was going to try and reproduce it, this is the best that I can do…
Bruce…
The problem with WinRot is that its a process that just seems to “happen” over a period of time. There’s no warning, no messages in the event log, no “Windows would like to rot now. Is this ok? Yes/No” dialog. Nothing. Today, its ok, in a months time my PC will use a lot more RAM to do exactly the same things, and in six months time it will use much more again. Eventually the machine just starts crashing randomly (usually Apps GPF, not full Windows BSOD’s), or some features just stop working, the thing starts behaving strangely, or going unresponsive at inconvenient times. Eventually this random behaviour drives you fscking nuts and you have to rebuild. Now I’m a programmer, I know things don’t “just happen”, but there doesn’t seem to be a specific event that causes it, the only constant is a long period of heavy use. The more installation/uninstallation of software you do, the less time it takes before the rot sets in.
Perhaps insted of agressively “dogfooding” every new beta of each successive OS internally Microsoft should run some machines to death in normal use then have the developers tear the OS apart bit by bit once it has gone rotten. Its real, and its getting worse, its been there ever since the registry was introduced in a big way in Win95. If I was a betting man, I’d be putting money on WinRot being some form of insidious registry corruption.
Of course, if the registry was a text file, I could keep it under source control and verify that for you, but I can’t, so I can’t.
Not much is it… Anyone want to add anything to what I’ve seen?
Scott is experiencing WinRot, that insidious disease whereby Windows starts getting wobbly and eats RAM like a pig after its been used for a while…
Full reboot and even a file system check. That’s two in two days. I know what I need to do:
Re-install Windows
I’m sure that would probably fix it — at least for a time. This machine is now 2 years old and that means that “winrot” has set in.
WinRot sucks, that it happens at all just isn’t good enough. The fact that it seems to get worse with each successive version of Windows over the years is criminal.
I’ve had the same sort of problems as Scott with Windows installs since Win 95. Over time they seem to rot, and just generally “go bad” for no real reason. The newer the OS, the worse this seems to get too (DOS/win 3.1 just didn’t rot, Win 95 slowly, 98 quicker, NT 4 took about 2 years to rot, 2000 went off in about 12 months).
I’ve got good news and bad news on XP though. I’ve been running it for 9 months on my desktop machine at work, and it hasn’t rotted significantly yet, thats the good news… The bad news is that the OS normally destroys itself catastrophically before it has a chance to rot! In those 9 months its been reloaded 2 or 3 times now. Even this morning my machine took 2 attempts to boot up, and I thought I was up for another reload. No error message, no blue screen, just rebooted mid way through the boot process. The number of times this machine has hard locked (no kb, no mouse, no power switch even) and I’ve lost everything since I loaded XP is unbeleivable.
Anyway, over the years I’ve rebuilt my desktop machines a lot, and I’ve got a few strategies which help…
First, you need two partitions… C: is for programs, D: is for data. Keep documents, code, MP3’s, digital photos, EVERYTHING on your D: (as in data) drive. Install Apps to drive C: and do nothing else with it. Blow away drive C: whenever you like, your data is safe on the other partition. Don’t use the default “My ” folders (My Documents, My Pictures, My Music, My Photos, etc XP has piles of them), as these usually end up on drive C: and violate this rule. Yes, you can change the locations of the “My ” folders, but thats usually too much hassle.
Next, on drive D: keep a folder structure with all of your drivers… Create a new folder for each OS version and device (eg. D:\Drivers\Win2kPro\Video, D:\Drivers\WinXPPro\Video, D:\Drivers\Win2kPro\Sound, etc). Whenever you download new drivers, extract them into this folder structure and install from there. When you blow away your OS, you’ve still got your driver tree ready for a reinstall. No looking for driver CD’s (which I’ve usually long since lost anyway).
Do something similar with downloaded program files. Create D:\Downloads and file all of your downloaded programs in it. If you download something and get rid of it, delete the installer and forget it. If you keep using it make sure you keep the installer. If you register it and it needs a serial number, create a text file in your downloads directory, and put the serial number in there so you’ll have it when you need to do a reinstall. Ideally give the text file the same filename (with a .txt extension) as the program installer so they show up next to each other in the explorer window.
Treat service packs, hotfixes, etc the same way, download ‘em once and keep ‘em (Windows update makes this harder, but less necessary). When you rebuild, you don’t want to wait while anything downloads.
Burn both D:\Drivers and D:\Downloads to CD periodically.
Finally, before you blow away your OS, log out and log in as the local administrator (or domain admin) user. Back up C:\Documents and Settings\Jim (or whatever your user name is) to your D: (as in data) drive. Its important to do this as a user other than youself so that you can copy the registry file thats sitting in there. If you run roaming profiles on your network, logging out will do this for you.
This folder structure is your user profile, and contains the HKEY_LOCAL_USER portion of the registry where most programs store therir settings, your browser histry, My Documents folder (which I still suggest you don’t use), etc. Copying that onto your new install can save you an enormous amount of time in reconfiguration later,
When you’ve loaded most of your apps, created your normal user account and logged in once to establish the new location of your profile, log in as administrator again and restore the backup. This will get you back all of your program settings, browser settings/passwords/bookmarks, etc.
Copying over the user profile over also gives you back your old start menu which can serve to remind you (via broken images where the icons should be) which programs you haven’t installed yet. Which helps to prevent that old “I used to have that program but I mustn’t have installed it again yet” problem that sometimes persists for weeks after a rebuild.
Hope that helps someone out… Reloading is a pain in the butt, but if you’re a little organised before hand you can get through it a lot faster… I usually rebuild my desktop in about 2 days, and my laptop in about 3 days (more drivers, more reboots).
I think I’ve now got my head in order enough to try and explain what I’ve been seeing with our client systems, and whats keept me too busy to blog for the last week or so… In the last week or so we’ve noticed a big increase in compromised machines within our client base. We’ve confirmed 7 machines compromised in the last week, which is a real worry. All machines run NT4 and are directly connected to the internet via a permanent dialup 56k modem. Most of these seem to be in Melbourne and Brisbane, where the sites would be on the same POP in each city, and therefore in a similar IP address range. Which suggests to me they’ve been picked up in a scan of a large block of IP addresses.
Normally, the first sign of a compromise is that the shared printers on that machine no longer function correctly, and this has an immediate impact on our software. One site has also observed that the machine appears to be under the remote control or an unauthorised third party at times.
Once “compromised” these machines seem to end up with a fair collection of tools installed, many of which I can’t identify fully, but the list includes…
Some machines also seem to have some sort of IRC bot, which suggests that they’ve been set up to be used as IRC XDCC file servers, or that they’ve been zombied for use in DDoS attacks. The IRC XDCC document offers a suggestion on how these machines might have been compromised, and what the machines might be being used for…
In this I noted the most common method people are using. Firedaemon will always be used, these hackers need to restart their programs each time you reboot. But, more sophiscated kids might employ other methods to scan or install these services. There are the ‘rootkits’, which are programs that auto mate the entire process. A hacker types in a range, and it will automatically scan for the vulnerability, copy files, run them, and secure they system, then move on.
On some systems, I can see logs which indicate that the machine has been used to scan another block of IP’s, usually looking for an open port 1433 (MS SQL Server). This is interesting, because scanning for port 1433 suggests that the “intruder” on my machine is getting in via MS SQL Server (which is installed on these machines), and is looking for other machines to compromise. However, the IRC XDCC hacking document referenced above suggests compromise via windows networking, not SQL Server. Maybe either is possible, but we need to look at securing both just in case.
When I’ve been looking at these machines I’ve been looking for files in c:\winnt\system32, c:\winnt\system32\drivers, c:\winnt\system32\drivers\etc, and c:\recycler which don’t belong. Files that don’t belong can usually be identified by a modification date which is more recent than the build date of the machine (in this case 31-1-01) which don’t seem to relate to any apps installed since the build (in this case that’s only Quicktime).
When you go into c:\recycler you’ll usually see two folders with recycle bin icons, 1 per usuer named with the users SID (i.e. S-xxxxx-xxxxx-xxxxx type file names). Anything else in the recycler directory is out of place.
So where to now??? First thing is to grab two machines and build them up from the ghost images we have and see if I can identify how someone might get into the machines. after that add hotfixes, service packs, and tune the security policy until I can’t get in anymore. Improving security policy probably means seeing if we can un-bind Windows Networking, and MS SQL Server from the dialup adapter, if they aren’t already. If anyone has got any other suggestions, please let me know!
I’d love to be able to shove the whole lot behind a Linux firewall or a dedicated router box, but thats not possible at the moment.
I’m blogging this mainly to help collect my thoughts on what’s been happening, and to solicit feedback from anyone who might have had similar problems recently. If this helps out anyone who’s seeing similar things, that’s great!
NewsForge speculates about what would happen if Microsoft open sourced Project.
Someone at Microsoft must realize that Office will eventually be cloned (OpenOffice is a start), ruining their market; if IBM can speculate that someday AIX may be dropped because Linux has grown up, wouldn’t it be better for Microsoft to enter the Open Source software industry and survive?
The article is however based on the assumption that the only way to compete with open source is to be open source. I’m not sure how valid that assumption is though.
I reckon the vast bulk of users (not geeks, users) couldn’t give a stuff about access to source code, because they’ll never need it. I’m more interested in seeing support for Open Standards become prevalent in proprietry solutions, than just Open Sourcing everything.
As an example, I’d like to see Microsoft adopt IMAP, LDAP, POP, etc as native protocols for Exchange, and phase out MAPI entirely. Allow access to the full callendaring and scheduling capabilities for the Exchange Server through open, documented protocols. Where those protocols don’t exist, create then. Where exisitng protocols are insufficient, work with standards bodies to extend them and support both old and new versions of the protocol. Open up the data store, allow migration from one platform to another. Don’t rely on “lock in” to keep customers, rely on your ability to deliver the best possible solution. That opens the way for companies to build a better server for Microsoft Outlook clients, or a better client than Outlook for Microsoft servers.
As a customer, I then have real choice. I can pick and choose which servers and clients I use, ad if any of them are any good, I might even pick the Microsoft versions. But without giving me the choice, its an either/or thing. I either go the Microsoft route, or I don’t, there’s less of a middle ground these days.
Arcterex (via Scott) bitches about Stupid, Stupid VB. I’m quite fond of VB (but thats not to say its not without its problems), but I have been using it for years, and some of his complaints are just plain wrong. So lets go through them one by one…
* If the program is running, you can’t edit the source code.
You can if you’re careful. Juch more than a trivial edit to the current line of code will force an application restart, as will changing a look construct wile you’re inside the loop. However, you can usually work around this by stepping ouside the loop, then working within the loop before stepping back in. Debug menu > Set Next Statement is your friend here (Alt-D then N). Sometimes its innevitable that whatever you change will require an application restart, I don’t know whether VB is better or worse in this respect than other languages, but I’d suspect its better than most.
* The help window’s last option when you right click on it in the task bar is not close, like every other window, but an ‘About HTML Help…’ option
This is annoying, absolutely.
* The home key will take you to to the end of the line if you are not there already, and then you have to hit it again to go to the start of the line. Isn’t that a bit counter-intuitive?
Pressing “End” takes you to the end of the line on my system. Pressing home takes you to the beginning of the code on a line (ie. first character after indenting), and a seconf press takes you to the beginning of the line before indenting. I actually like this behaviour, because most of the time if I hit Home, I want to change something at the beginning of the line, and don’t then want to arrow over the indenting to get there. Of course if I need to play with indenting, its a simple “Home” press away from there. I find working in editors that don’t do this a bit frustrating at times actually.
Are you sure you’re running the right keyboard driver/language for your system? I’ve never seen the Home key take you to the end of a line in ANY microsoft product.
* When the program you write has a modal dialoge box popped up (say, for debug purposes), you can’t switch back to the IDE and stop the task! So if you’re doing debugging where a message box is up on every iteration of say, a loop, you have to be really fast to hit the “OK” button and then hit “stop” in the IDE.
Pressing Ctrl-Break at any time will stop Execution of a VB program running in the IDE. So when the message box appears, hit Ctrl-Break then click on Ok, Yes, No, Cancel, whatever. The current line of code will finish execution and return (in this case the MsgBox call will finish execution and return when you click on a button), then the IDE will break before running the next line. No keyboard/mouse speed tests required. Using this trick I sometimes do stuff like…
If InIDE Then
MsgBox "Debug From Here"
End If
.. To create a persistent breakpoint, because your current breakpoints will be lost when you save the file and quit VB. The InIDE function is a dead simple function I found a while ago to detect when the program is running under the IDE.
* If you’re not fast enough, you can always go to the task manager and kill the application (not the IDE). Oh wait, the two are tightly tied together, so now you killed your development environment as well!
This is annoying, I agree. Occasionally, you get a runaway application that you just want to stop, and you can’t use Task manager without killing the IDE as well. Theres two solutions here, and I use both. 1) Turn on the option to save the code before running it. I can’t count the number of times this has stopped me from loosing work when VB crashes, or I task manager VB while the app is running. Just Do It, you won’t regret it. 2) Ctrl-Break. Even when the app is in the middle of a tight loop madly doing something, and the IDE won’t respond to a click on the stop or break buttons, Ctrl-Break will work. The current line of code will finish execution, then the IDE will break before running the next line. This can sometimes lead to a delay before breaking (perticularly on large/expensive database operations), but it always works. I haven’t had to Task manager VB for a long time.
* The help system is nice, but if I say I only want to search in the Visual Basic Documentation only (via the “Active Subset” drop down), I mean it! Don’t give me other topics, only what I ask for. Another fine example of Microsoft giving the user what they think the user wants, not what they asked for.
The “Visual Basic Documentation” subset includes things *most* Visual basic programmers would use, such as ADO, etc as well as “pure” VB documentation.
* Ok, looking back at the above bitch about the help search not using my filter, it is, kinda. It’s restricting from some documentation, like the really extranious stuff, but “Visual Basic Documentation” also searches IE, MS ADO, Active Server pages, and a bunch of other stuff that seems loosely tied to VB. Why can’t I search just the Visual Basic Reference?
As I said, *most* (you may not be a typical VB developer, I’ll grant that), would use these additional topics. However, I created a subset of my own long ago that included just what I use in that subset. Go to the View menu, then Define Subset and create your own from there.
* If you’re going to ignore my topic to search in the help, at least save the sort field of the results so I don’t have to resort after every search.
Granted.
* And why can’t I open more than one help window at once? What if I want to look up two things at the same time, and compare them or switch back and forth?
I can’t say I’ve ever needed to do this.There are Back and Forward buttons on the viewer, so that might help. also, the entire documentation collection is available on MSDN Online, so you always have the option of using the installed version for context sensive help, then keep multiple browsers open for the rest.
* Who thought up the “rank” system for the search by the way? I search for “MsgBox Function” and the page titled “MsgBox Function” gets a rank of 17, while “Designing an error handler” gets 1. “Change Event” gets 111 (it’s the 111th result), and I’m not sure which way the scale of “rank” goes. They need to take a look at google I think.
The ranking system sucks badly. If I know the VB function I’m looking for (eg. MsgBox) I usually just type it in and hit F1 to get help rather than searching for it. Microsoft search tools by and large suck, the MSDN Library is no exception. Sometimes a Microsoft Google search will turn up better results, and should include MSDN Online. I wish MS would give up on writing their own search tools and just embed Google.
* Why doesn’t my scroll mouse scroll? It does in every other app I’m using right now.
I’ve never had a scollie mouse, so that doesn’t bother me. It does however, probably relate to the age of the product. Remember that for standard controls scoll wheel support is automatic, but if you’re coding your own control (as Microsoft would have to do with the VB code window), you do need to explicitly code for it, or inherit the behavious from the control you’re building on. VB6 was released in 1997 and I don’t think scroll wheels existed back then, so its a bit hard to explicitly code for something that doesn’t exist.
Granted, Microsoft could have added scroll wheel supportin a service pack, but they’ve been criticised for adding features in Service Packs before (anyone remember NT4 SP4?), and now have a stated policy of not doing that.
I have found other little editing oddities on the VB code window though, like Shift-Arrow key highlighting doesn’t work when operating VB remotely via VNC, and the search and replace window doesn’t centre itself properly on a multiple monitor system (it wants to centre across monitors, and ends up half on each).
I’m not trying to say that VB is perfect, but I just wanted to point out that some of these criticisms are unfounded. Theres actually a pile of things that annoy me abou VB, maybe one day I’ll get around to writing them up.
A customer of ours has fallen victim to the jdbgmgr.exe virus hoax, and deleted the file. This page from the MSKB explains how to recover the mess.
You wouldn’t accept medical advice from some guy in the street, so why on earth people take IT advice from anyone who emails them I don’t know!
Simon Fell has released PocketHTTP v1.0.0 an open source COM component for HTTP transport. Looks good!
SecurityFocus does an excelent job of explaining the real facts behind the recent Windows XP vulnerabilities.
News flash: this is expected, and desirable, behavior. The Win2k RC can’t read the XP registry, so it thinks it is a corrupted Win2k installation. When it can’t verify the SAM, it bails out to the console. Administrators want this behavior. If you have an installation on which some third-party driver has hosed the registry, the Recovery Console will allow you to attempt to fix it. That’s what “Recovery Console” means.
In this article The Reg brings us the news that SQL Server developers could be up big $$$ after a licencing dispute.
A Washington court ruling could see SQL Server developers liable for millions of dollars in licensing fees.
Timeline, who’s patents have been involved in the dispute have posted more details on their web site.
Timeline takes the position that Microsoft Analyst Services databases built with Microsoft’s tools (Manager) provide all necessary steps to infringe one or more of the independent claims of the ‘511 patents. It would follow that third party products which provide the additional material feature or function covered by a dependant claim to one of those independent claims would cause a new infringement outside the scope of Microsoft’s license. Microsoft does not concede that its products in fact infringe, nor was such a statement required under the Timeline patent license it procured. Also, not every user of SQL Server will use the allegedly infringing portions of SQL Server. However, that is now moot. A combination where all material steps are present, and at least one material step is performed by a third party, requires its own license; regardless of whether Microsoft itself provided sufficient steps to independently infringe a different claim.
While I’m not a lawyer, it would appear that because our products don’t use Data Transformation Services (DTS), we’re not affected by this. Thank god I decided to implement our own product file import facilities and not use DTS to do the job.
