Neil Schemenauer's Web Log


April 26, 2017

Normal CDF in Python

I needed a single function provided by scipy, i.e. scipy.stats.lognorm.sf(). I think scipy is a high quality library but having to include such a huge library just for a single function bothers me. So, I did a bit of research as to how to implement it myself. I need an accurate and high performance version and my initial table-based linear interpolation version was not accurate enough.

It turns out that modern versions of Python have the ERF function included. That gets you most of the way there. Computing the normal (Gaussian) CDF is pretty simple.

def norm_cdf(x):
    '''Probability that a statistic is less than x. This equates to the
    area of the distribution below x.
    return 0.5*(1.0 + math.erf(x/math.sqrt(2)))

What my algorithm actually needs is the log-normal survival function, but getting to that from the normal CDF is also pretty simple. Traditially, the log-normal distribution takes two additional parameters, mu and sigma (i.e. location parameter, scale parameter). The survival function then is as follows.

def lognormal_sf(x, mu, sig):
    return 1 - norm_cdf((math.log(x)-mu)/sig)

Trivial test:

>>> scipy.stats.lognorm.sf(1, 2, 0, math.exp(.5))
>>> lognormal_sf(1, .5, 2)

Thanks to Mark Dickinson for adding math.erf() to Python. Implementing those special functions accurately is non-trivial and the versions in Python look to be high quality.

July 06, 2013

Bridging with 2wire 2701HG-S DSL modem

Sasktel has installed a 2wire 2701HG-S to provide DSL service for me. After quite a bit of messing around, I believe I've figured out how to make it act like a bridge instead of routing with NAT. The instructions from James Hartig on how to use the DMZplus feature are key.

The short version is to use the firewall menu to mark your own router as the DMZplus device. As far as I can tell, the 2wire device uses the MAC address of the second router to bridge traffic (layer 2 bridge?). Your router can use DHCP to get the WAN IP address. I care about latency so I don't want two layers of NAT. The 2wire device is fairly flexible but not like OpenWRT or DD-WRT, etc.

July 28, 2012

Making of Warcraft

A great post about working on Warcraft. Aside from interesting background on development details, I'm struck by the incremental process of innovation. I never played Dune 2 but I had heard that it was a pioneer for RTS games. Patrick Wyatt confirms the influence. Also, the influence of Warhammer is noted. I don't think Blizzard would ever admit such things, Starcraft/Warcraft is their creation and they did it all themselves and you better not infringe on "their IP".

I intensely dislike the media's tendency to portray each technological advance as the result of one or a couple of people's genius. Except in very rare cases, progress does not work that way. Instead, more often than not, a "breakthrough" is actually just a small improvement on an existing idea. Also, often the same improvement is "discovered" multiple times completely independently. It seems like when an idea's time has come, it's discovery is almost inevitable.

Anyhow, enough ranting. I enjoyed Patrick Wyatt's blog. If you are interested in game development, check it out.

July 05, 2012

Where's Chigurh?

On some weird tangent, I started thinking about No Country for Old Men again today. It's one of my favorite movies. I enjoy almost anything by Coen brothers and the acting in the movie was also fantastic. Like many good movies, and like a good Rorschach ink blot, the details of the plot are left open for interpolation by the viewer.

One key point of the plot left ambiguous is: where is Chigurh when Ed Tom enters the hotel room?. There are lots of theories out there. I've never seen my theory presented however, so I feel like I should write about it.

To me, the central theme of the move is Ed Tom's feeling that he is too old to continue fighting evil. He feels like crime is getting worse and he is no longer up to fighting it. He's "overmatched". Chigurh represents the modern evil. The meeting in the hotel room is the ultimate test of Ed Tom. Is he up to it?

To me, it seems clear that Chigurh is in the room. There is no good reason as to why he would not be there and that's why it was a popular topic of discussion after the movie's release. So, if he is there, why was there no confrontation?

The key clue is provided when the accountant asks Chigurh "Are you going to shoot me?". Chigurh responds That depends. Do you see me?. There are numerous examples in the movie of evidence that Chigurh lives by some strict code of rules.

Ed Tom makes himself not see Chigurh. It's the only explanation that makes sense to me. Chigurh is standing behind the door but Ed Tom ignores him. He is not up to confronting the evil. He knows he will die and he is afraid. Chigurh follows his code and does not kill him.

The exact details of how the situation unfolds is not clear. I haven't watched the movie in a while but in my imagination, Ed Tom initially does not see Chigurh. When he does, Chigurh has got his gun pointed at him and there is no way Ed Tom could win the showdown. Instead Ed Tom lets Chigurh escape.

At the end of movie, the first dream is about this failure. Ed Tom's father entrusted him to uphold the law, protect innocent people, fight against evil. Ed Tom failed. The second dream is a forgiveness for this failing.

April 03, 2012

Samsung Galaxy S (I9000) partitioning scheme

I had trouble finding details about how flash memory is partitioned on this device, here's what I learned in case it helps others. The device has two built-in flash memory devices: 8 or 16GB NAND and flash memory connected directly to the processor chip (MMC).

The MMC can be partitioned using different schemes. The stock ROMs use a scheme called BML. Most open source ROMs use a more flexible scheme called MTD. When you first install CyanogenMod 7, for example, the MMC is converted to use MTD partitioning. The MMC partitions are as follows:

If you are familiar with the Heimdall or Odin tools, these names will be familiar. Odin and Heimdall use the PIT file to control the partitioning of the MMC (e.g. s1_odin_20100512.pit).

The NAND device is partitioned into two parts: /data and /mnt/sdcard. The partition mounted on /mnt/sdcard is referred to as the internal SD card. If you put a SD card in the external slot, it is mounted as /emmc (at least in CM9).

For reference, here is the NAND partition information for my i9000m phone.

klonk:~$ adb shell
shell@android:/ $ su
shell@android:/ # fdisk /dev/block/mmcblk0

The number of cylinders for this disk is set to 1953792.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/block/mmcblk0: 16.0 GB, 16005464064 bytes
1 heads, 16 sectors/track, 1953792 cylinders
Units = cylinders of 16 * 512 = 8192 bytes

              Device Boot      Start         End      Blocks  Id System
/dev/block/mmcblk0p1               5     1708032    13664224   c Win95 FAT32 (LBA)
Partition 1 does not end on cylinder boundary
/dev/block/mmcblk0p2         1708033     1953792     1966080   c Win95 FAT32 (LBA)
Partition 2 does not end on cylinder boundary

Command (m for help): 

On my phone /data is formatted as ext4 and /mnt/sdcard as vfat. One other thing, if you connect your phone to a computer with USB, the mass storage devices that are visible correspond to /mnt/sdcard and /emmc (i.e. one partition of the NAND and the external SD card). The other devices are not visible.

March 27, 2012

Microsoft Exchange time-zone confusion

This has happened to me more than once and somehow I think I'm not the only one. When Microsoft Exchange sends out calendar related email, it includes time-zone information in the text of the message. Good idea, right? Well, the implementation leaves something to be desired. For example, if the message contains the line:

When: Wednesday, March 28, 2012 3:00 PM-4:00 PM (GMT-06:00) Central Time (US & Canada).

what time would you guess the event occurs at? If you guessed 21:00 UTC (12+3+6) then you lose. The time zone offset is actually GMT-5 as the "text/calendar" attachment (not so) plainly shows. The problem is that the local time-zone of the sender was Central Daylight Time.

February 20, 2012

Silly Quixote benchmark

Looks like the old dog can still run with the young pups. If you are starting a new web project in Python, Flask is worth a look. I spent some time poking around at the source code and it looks decent. The "context" hacks are probably the ugliest part of it but not so bad. I don't care for Django, it's origin as a content delivery system is pretty apparent.

BTW, Durus has been doing NoSQL for ages, almost as long as ZODB. :-)

May 02, 2011

Hooray for paper

I like technology. Sometimes though the old ways are still best. I voted in the federal election this morning. I used a graphite pencil to make a mark on a slip of paper. It really was about as painless as possible. The simplicity of system greatly reduces the chances of fraud or failures.

I'm heading to the field soon. There exist a number of field record keeping systems based on PDAs, mobile phones or computers. For the foreseeable future I plan to continue to use a per-printed paper form, a clipboard and a pencil. Entering the data into the computer after the fact is not a serious problem. I'll let someone else be the technical pioneer in this case.

Neil Macdonald reports on Osoma's death

"The Devil likely died happy".

I hope American citizens will continue to fight to restore lost liberties. If you haven't seen it, The Power of Nightmares seems fitting at this point given that some of the news reporters are asking what this means for the leadership of al-Qaeda.

Why We Fight is also worth watching. The notion that terrorists attack because they "hate America's freedom" is ludicrous. To effectively fight someone you really should be clear on their motivations.

November 26, 2010

Sasktel's new 3G+ network

Sasktel has been recently bragging about their new "3G+" network. For a change, it actually is something worth bragging about. Cell phone technology is extremely complicated. There is an old saying that the good thing about standards is that there are so many to choose from. Cellular network people have taken that idea to a whole new level.

Johannes Lindenbaum has a pretty good explaination of the situation. In summary, Sasktel, Telus, and Bell are all deploying UMTS/HSPA networks that use the 850/1900 MHz bands. New phones will have SIM cards and so you can switch phones just by moving the SIM card. People in other countries had this feature with their GSM phones for long time already (many places it is required by law). We North Americans are a little slow.

The buzz is that Sasktel is selling SIM cards to be used with unlocked phones. This is much bigger news than the supposed superior data speed of the "3G+" network, IMHO. People have already been buying iPhones elsewhere and then buying a $10 SIM card from Sasktel to activate them. Obviously you still have service fees but being able to buy and sell phones and easily move them between providers is a huge deal.

Want to buy a phone to use on the Sasktel 3G+ network? Minimally, it must support UTMS on 850 and 1900 MHz. That will give you voice service. Watch out, some phones support GSM on 850 and 1900 and UTMS on 900 and 2100 and sometimes the labeling can be confusing. Most UTMS/HSPA networks in the USA use the 900/2100 bands and those phones will not work. AT&T uses 850/1900. The phone should support HSPA if you want data services (I'm not sure about the difference between HSPA, HSPA+, and HSDPA). The phone needs to be "unlocked". Because phones are often sold at a loss, companies lock them to a single SIM card so they have to be used on a certain network. Courts have ruled that it is legal to break this "lock" (usually you just need a numeric code) so you can use any SIM. If the phone does not come unlocked you can usually pay someone to unlock it for about $20.

One final note, you don't need a "smartphone" (another marketing term) or a data plan to get on the 3G+ network. All you need is a phone that supports UTMS on the 850/1900 Mhz bands. Finally take this information with grain of salt. I'm not an expert so do your own research before spending big money on something like a Nexus S. ;-)

[comments] [validate]