Private (from Facebook) browsing

Facebook has been compromised by a clickjacking exploit, again.

Has anyone made a Firefox extension or Chrome plugin to load Facebook links in private-browsing window, and keep the user logged out while browsing their main window? I think that would solve all this clickjacking and Facebook-web-spying.

Data-mining in the Uncanny Valley: An e-Commerce Puzzle

Here's a "psychology of data mining puzzle" for you.

I received the following email from an e-commerce site. (Names expunged to protect the innocent and guilty.)

Dear [Customer],

One year ago, you ordered the following product from [e-commerce site].

[So-and-So] Salad Bowl & Servers - Stone - One Size/One Size

We wanted to let you know that right now, your size is still available
from [e-commerce site]. You can order the same product again by visiting:

http://www.example.com/bin/z/ref/oneyear/...html

Or, if you'd like to view our entire [So-and-So] collection, please visit:

http://www.example.com/bin/z/ref/oneyear/...html

If there's anything we can do to improve our service, please don't hesitate to let us know!

Why would the e-commerce site think I'd want to buy the same product again one year later?

What do you think was the rationale for this email campaign?

Here is an important hint, if you want one. Stop reading, if you don't want a hint. (The e-commerce vendor in question has a name that start with zap and ends with pos, and has a particular area of focus.)

Privacy Bugs in Picasa's Batch Upload

 

Privacy Vulnerability in Batch Upload

I have been using Picasa/PicasaWeb  to share photos. Here is my workflow:

1. Load all photos from my camera onto my computer, creating a folder for each event/day.
2. In Picasa, "Star" my favorite photos.
3. Sync the folder to web, "Starred photos only", and share that folder publicly.

Today, I discovered the Batch Upload feature, and thought it would be a good way to create a private online backup of all my photos. Mistake!

When you use the "Batch Upload" feature in Picasa, you choose Online settings, like Photo Size, Visibility, and Sync On/Off. If you are Uploading a folder that is already partially synced to Web, these settings are ignored.

If you have already chosen to Sync the Folder to Web, the new upload is merged with the old upload. Previous settings are maintained, including visibility. Picasa's Batch Upload will set the entire folder to Public, even if you choose "Private" in the Upload Options!

There is a separate radio button for "Change options".  If you are using the Upload to create a private backup,  Use the "Change options" button to make an album private.

However, doing so will make ALL the photos private, including the Starred Photos that were previously public. So, you will then need to create new albums for photos you want to share.

If you have already shared a folder in the past, you'll need to go back and update blog posts / web links to point to the new albums. 

Effectively Batch Upload gives you two options:

1. Publicize ALL your photos in a folder,
2. or Privatize ALL your photos.

There is no way to create an online folder (album) that is part public and part private.

 

If you want to use "Batch Upload" as a Backup feature, you can't use Folder Sync (with Starred Photos only) as a "sharing/publishing" feature.albums.

Pooh!

Connecting to Postgres from R on Windows 7 64bit

Here is some quick advice on how to get up and running with R + Postgres on Windows 7 64bit. There are several possibilities, but most have fatal flaws. Here is what worked for me:

Add jvm.dll to your PATH

rJava, the R<->Java bridge, will need jvm.dll, but R will have trouble finding that DLL. It resides in a folder like

C:\Program Files\Java\jdk1.6.0_25\jre\bin\server

or 

C:\Program Files\Java\jre6\jre\bin\client

Wherever yours is, add that directory to your windows PATH variable. (Windows -> "Path" -> "Edit environment variables to for your account" -> PATH -> edit the value.)

You may already have Java on your PATH. If so you should find the client/server directory in the same Java "home" dir as the one already on your PATH.


To be safe, make sure your architectures match.If you have Java in "Program Files", it is 64-bit, so you ought to run R64. If you have Java in "Program Files (x86)", that's 32-bit, so you use plain 32-bit R.

Re-launch R from the Start Menu

If R is running, quit.

From the Start Menu , Start R / RGUI, RStudio. This is very important, to make R pick up your PATH changes.

 

Install rJava 0.9.2.

Earlier versions do not work!  Mirrors are not up-to-date, so go to the source at www.rforge.net: http://www.rforge.net/rJava/files/. Note the advice there

“Please use

install.packages('rJava',,'http://www.rforge.net/')

to install.”

That is almost correct. This actually works:

install.packages('rJava', .libPaths()[1], 'http://www.rforge.net/')

Watch the punctuation! The mysterious “.libPaths()[1],” just tells R to install the package in the primary library directory. For some reason, leaving the value blank doesn’t work, even though it should default.

Install RpgSQL.

install.packages('RpgSQL')

 

Set Classpath. 

Set a classpath to use for pgSQL:

jdbcClasspath = "C:\\Program Files (x86)\\PostgreSQL\\pgJDBC\\postgresql-8.4-702.jdbc4.jar"
);

Do NOT use .jaddClassPath(). The pgSQL method ignores .jclassPath()

 

Create a DBI driver.

The Examples code in help("pgSQL") has errors (as does most R documentation, sadly). Here is how to set up your driver instance:

myPgSQL = pgSQL(driverClass='org.postgresql.Driver', 

                jdbcClasspath, 

                identifier.quote="\"");

Make a Connection.

connection <- dbConnect(myPgSQL,
  user = "your_username",  
  password = "your_password", 
  dbname = "your_db_name", 
  host = "your_db_host", 
  port = 5432
);

Query!

dbGetQuery(connection, "select datname from pg_database");

dbGetQuery(connection, 

  "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'");

# dbGetQuery returns a Data Frame, very nice.  

result = dbGetQuery(connection, "select * from markets"); 

View(result);  

head(result, n=1)  # Show 1 just row of data

head(result$name, n=1)  # Show 1 just row of data

help("pgSQL")    ; # To see more example queries

dbDisconnect(con); # When you are done.

Let me know how it goes for you!

Linux virtual desktop: “Remote” audio-video access

“The post documents how I set up my Linux desktop system. My particular system runs inside VirtualBox on Windows, but most of the discussion here applies to any Linux productivity desktop.
You may wonder why I am running on VirtualBox. There are a few reasons:

1. With VirtualBox, I can run a Windows system and a Linux system on the same piece hardware at the same time.
2. Windows has much better hardware driver support than Linux, especially around power management.
3. VirtualBox provides virtual hardware to the Guest Linux OS, and this hardware is much better supported by Linux than real hardware.

Headless Linux

Linux has a client-server architecure that allows us to run a program on one machine that sends its output to another machine. “Output” includes text, and also graphics and sound!

When we run a program without using the video/audio/text console of the machine where the program is running, we say the program is running “headless”. To make use of such a program, we run a video/audio/text server on a different machine (usually the one where we are sitting, which is not where the program is running).

This section explains how to run Audio and video servers on Windows, for use with programs on remote Linux systems.

Video Setup: XWin video server

You’ll need Cygwin/X to make graphics work.
Run XWin Server on your local/host machine, using the Shortcut or the command:

C:\Program-Files\Cygwin\bin\run.exe /usr/bin/bash.exe -l -c /usr/bin/startxwin.exe

Set and export DISPLAY (It’s best to due this in ~/.bashrc, so it runs automatically in every shell session):

export DISPLAY=:0

eSound audio server

We need to install a sound server on Windows, to play sounds sent back from the remote Linux system. Esound and Cygwin come to the rescue:

Run Cygwin setup.exe and choose to install “eSound”.

Then, run eSound (in a loop, in case it dies):

while true ; do esd.exe -tcp -public ; sleep 5; done

You’ll probably want to stow this command somewhere that will make it run exactly once each time you boot your Windows machine.

Connecting your Headless system

Use ssh to connect to the Linux machine.

• Add the –Y option to enable X forwarding, so you can see the GUI of guest OS programs on your local desktop.
• Add the –R option to set up a tunnel for eSound

ssh –Y -C -R 33001:localhost:16001 linux_user@linux_machine

When you log in to the Linux machine, instruct your session to send eSound output over the network through your tunnel.

export ESPEAKER=127.0.0.1:33001

Run your favorite programs with graphics and audio, like xterm or even firefox.  You’ll need to instruct any program you run to send its sound to eSound. Consult the documentation for each individual program.
For example:

ESPEAKER=127.0.0.1:33001 FIREFOX_DSP="esddsp" firefox

(This works using Cygwin/POSIX. You might need to tweak things to run from a Windows command shell.)
Reference: http://www.gbar.dtu.dk/wiki/Remote_Sound#1._Manually_by_SSH

Setting up Linux on Windows with VirtualBox

 

After trying and failing to run Ubuntu as my primary OS (for reasons I am not in the mood to discuss, I relented and installed Windows 7 Professional. (I bought a heavily discounted copy through Friends and Family program at the Microsoft company store. Never pay retail price.)
 
I use VirtualBox to host a virtual Linux machine inside my Windows Desktop. This setup lets me to have a (vaguely) usable Desktop console experience, with sleep/wake and power management, wireless, stable desktop graphics, and all those nice things.

This article describes how I set up VirtualBox to host Ubuntu as a Guest OS running inside Windows. A separate article will discuss how I set up the purely-Linux bits. This article discusses how to install and configure Virtual Box, and how configure the Linux<->Windows connection.

Install VirtualBox and create a virtual machine.

Install VirtualBox 4.1.2

Create a new Virtual Machine, following the prompts.

I like to separate my large “special” files from the rest of my home directory. This makes backups easier – I can frequently backup my small set of personal files, and backup large media/system files on a different schedule.

So, change the default locations for the Virtual Disk Image (VDI) and VM Snapshot, since these are very large files. I use C:\data.

You’ll need to install an OS on your virtual machine. I use Ubuntu. In VM Settings –> Storage –> IDE Controller, click on the CD icon on the right and choose a virtual CD/DVD (ISO image file) to “insert” into the virtual optical disk drive.

Start your VM, and it will boot from that “virtual CD” and let you install an operating system (called the Guest OS on your PC, distinguished from the Host OS, which is Windows).
Once the Guest OS is installed, I strongly recommend that you install an SSH server as soon as possible, so that you can connect to your virtual machine without using the virtual display, which is memory-intensive and CPU-intensive.

Guest Extensions

Guest Extensions is a package of software that runs inside the Guest OS, to enable special Virtual Box features. Generally, these are “virtual drivers” that let the Guest OS communicate with VirtualBox, enabling great features like fancier mouse/keyboard support, higher/variable “virtual display” resolution, and exposing host system directories as virtual hard drive partitions in the guest.

After installing and booting the OS, install “Guest Extensions”. To do this, pick the option from the Menu at the top of the window in which the Guest OS is running. That will mount a virtual CD. Run the “autorun.sh” shell script (as root) from that virtual CD, and Guest Extensions will install. Guest Extensions fixes GUI resolution, making the Guest OS display size match the window size, which you can change by clicking and dragging on the corner of the window.

After installing Guest Extensions, shut down the Guest OS, so we can finish setting it up.

Extension Pack

Extension Pack is different from Guest Extensions.  Extension Pack is a package of Host software that extends the functionality of VirtualBox. Extension Pack has the non-Open Source code of Virtual Box, which is why it is packaged as an extension.

The most important Extension Pack feature is USB 2.0 filter support. This feature lets your guest OS interact directly with your Host machine’s USB 2.0 devices. Download and install Extension Pack, and then visit Settings –> USB in VirtualBox Manager. Here you can check the Enable USB 2.0 box, and then click on the “USB-plus” icon to choose devices you want to provide to the Guest OS.

One very important USB 2.0 device is our external hard drive, which has a Linux formatted (ext-format) partition that Windows cannot see.

Networking

VirtualBox uses NAT (network address translation) networking by default, which is (probably) the same setup that your home PC uses to connect to the Internet. This is a one-way sort of networking:  It’s easy to reach the Internet from the machine, but hard to reach the machine from the Internet. In the case of a virtual machine, your local network is considered part of the Internet. This is probably not want you want.

Most likely, you’ll want to connect to the Guest OS via ssh, or a webserver, or some other network protocols, besides a direct virtual console (keyboard/mouse/display) connection. To do this, you’ll need to set up either Bridged Networking or  Port Forwarding.

If you want to treat your VM like a real machine on your local network, so that you connect to services (SSH, HTTP) running on the machine in the “usual” way,  then you’ll want to set up Bridged Networking, which basically instructs Virtual Box to run a virtual switch to direct traffic between the virtual machine (one one side) and the Host machine and rest of the local network network (on the other side)

Bridged Networking

While our virtual machine is powered off, open VirtualBox Manager and navigate to Settings –> Network –> Adapter 1. Change the “Attached to:” option from NAT to Bridged Adapter. Make sure Cable Connected is checked.

Now you are set up for full bidirectional network connectivity on your virtual machine. Have fun!
Also, beware: Your virtual machine is now equally exposed to the Internet as your host machine, so be conscious of security.

In particular, if you follow the (very wise) practice of running a web browser inside a virtual machine, in order to isolate your PC from malicious web sites, then you need to turn off Bridged Networking on any virtual machine that you want to quarantine from the Internet.

Port forwarding

Skip this section if you have chosen Bridged Networking.

I wrote a simple shell script (requires Cygwin to run) to set up Port Forwarding from the Windows desktop to the VirtualBox guest Linux system. This script sets up forwarding from Windows:2222 –> Linux:22 for ssh, Windows:8787 –> Linux:8787for Rstudio, and 8000 –> 80 for http. I hope it’s obvious how you can modify it to set up the services/ports you need.

I named this script virtualbox-port-setup. Run it like so:
  virtualbox-port-setup VM_Name

#!/bin/bash

test "$1" == "" && echo "USAGE: $0 guest_name" && exit 1
guestname=$1
net_path="VBoxInternal/Devices/pcnet/0/LUN#0/Config"
function run() {
    echo "$@"
    "$@"
}
function forward_port() {
    cmd="VBoxManage setextradata $guestname"
    local service=$1
    local host_port=$2
    local guest_port=$3
    local protocol=$4
    
    run $cmd "$net_path/$service/HostPort"  $host_port
    run $cmd "$net_path/$service/GuestPort"  $guest_port
    run $cmd "$net_path/$service/Protocol" $protocol

}
case $2 in
    undo)
      # unset settings
    forward_port ssh 
    forward_port rstudio 
    ;;
    *)

    forward_port ssh 2222 22 TCP
    forward_port rstudio 8787 8787 TCP
    forward_port http 8000 80 TCP
    ;;
esac
    
echo
echo "Updated NAT config:"
run VBoxManage getextradata $guestname enumerate | grep $net_path

(Side note: I am using Code Formatter Plugin for Windows Live Writer to insert code snippets into this article.)

You can run this script at any time, but you’ll need to completely close down your Guest OS (I think saving state is fine) in order to complete the setup and activate port forwarding.

Important! Activating port forwarding breaks your virtual machine. (There is a fix.) After closing the virtual machine, open VirtualBox Manager, edit Settings for the VM, navigate to the “Network” section, and change “Adapter Type” from Intel to PCnet-FAST III Am79C973. This change will fix the error “Configuration error: Failed to get the "MAC" value. VBox status code: -2103 (VERR_CFGM_VALUE_NOT_FOUND).”



If you ever decide to remove the NAT, run this script with the “undo” argument, (or else your virtual machine will fail to start up, giving an error about “unknown configuration node”):

   virtualbox-port-setup VM_Name undo

Running Headless

You could launch the  virtual machine now, but here’s a better idea: Run the VM headless. Headless VM has several benefits:

1. You can save the memory and CPU cost of running a graphical desktop environment inside the Guest OS.
2. You can use a host-side (Windows) X Server that will let you run each of your graphical guest/Linux programs “rootless” on the Windows desktop, instead of all locked inside a single Linux container window.

To manage your headless guest OS, use these commands:

# start
"C:\Program Files\Oracle\VirtualBox\VBoxHeadless.exe" --comment "<VM name>" --startvm "00000000-0000-0000-0000-000000000000"

# suspend
"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm  "<VM name>" savestate

You’ll need to change the <VM NAME> and GUID (long string of digits) to match your VM. An easy way to get this information is to enter VirtualBox Manager, right-click on your VM entry, and choose “Create a Shortcut on Desktop” Then right-click on the shortcut, choose Properties, and copy-paste the VM specs.

 

Connecting your Headless VM

Remember when I told you to install an ssh server? I hope you followed my advice. We are going to connect to the virtual machine over ssh, and forward graphics and sound back to the local (Host) machine.

Before we connect, we need to prepare the Host system to play audio and video from the guest system.

Simple: Audio Setup: Use your head

The simplest way to play sound is through the “Head” console.

To do this, start your virtual machine in the “normal” way (non-headless), and login to the desktop session.

Then, any programs you run (on the head console or on a headless connection) will send audio to the head console for VirtualBox to play for you.

The idea here is that you can still run rootless GUI windows on your Host system, while playing audio through VirtualBox.

Running the head console loses some of the RAM-savings and CPU-savings of pure headless mode, so you’ll want to configure your virtual machine’s desktop session to be a minimalist lightweight desktop that can still play audio.

If you want to run totally headless, follow the advice for “Headless Linux” in the Linux article in this series.

Now we are ready to connect!

If you are using NAT Port-Forwarding: To connect to your VM, ssh to the forwarded port on the host machine, using the username and password of an account on the guest OS.

If you are using Bridged Networking: To connect to your VM, ssh to the guest machine hostname/IP, using the username and password of an account on the guest OS.

• Add the –Y option to enable X forwarding, so you can see the GUI of guest OS programs on your local desktop.
• Add the –R option to set up a tunnel for eSound

Port-forwarded NAT networking:

    ssh –Y -C -R 33001:localhost:16001  guest_os_user@host_machine –p 2022

Bridged networking:  

    ssh –Y -C -R 33001:localhost:16001  guest_os_user@guest_machine

(This works using Cygwin/POSIX. You might need to tweak things to run from a Windows command shell.)

Raw hard-drive access

I have a Linux-formatter hard disk from my machine’s briefly previous life running Linux. I would like to access that disk directly from Linux. Here is how we do it:

WARNING: Do not attempt to mount any disk/partition that is also mounted by the Host system, especially not the Host boot disk!

WARNING: Some programs, like Nautilus, will auto-detect all partitions on a hard drive, and attempt to mount them if you click on an icon. Clicking the icon for any partition mounted by the Host OS will crash the Guest OS!

Create VMDK files. Here is a Cygwin/bash script that can help with that.

drive_infos=$@for drive_info in $drive_infos ; do
        drive_number=$(echo -n "$drive_info" | sed -e 's/:.*$//')        partitions=$(echo -n "$drive_info" | sed -e 's/.*://')        if [ "$partitions" == "" ] ; then                partitions_flag=""        else                partitions_flag="-partitions $partitions"        fi        vmdk_file=$vmdk_dir/RawDisk-PhysicalDrive${drive_number}-$partitions.vmdk        echo_run VBoxManage internalcommands createrawvmdk -filename $vmdk_file $partitions_flag -rawdisk '\\.\PhysicalDrive'$drive_number

In VirtualBox Manager, Settings –> Storage, add the newly VMDK files as virtual disks. (Check “Solid State” if appropriate.)



Use  “listpartitions” to find the partitions numbers on your disks, if you want to only expose non-Windows partitions to Linux. Then you can pass drive numbers like “1:3,4” to the script above, for drive 1, partitions 3 and 4):

VBoxManage.exe internalcommands  listpartitions -rawdisk '\\.\PhysicalDrive0'

VBoxManage.exe internalcommands  listpartitions -rawdisk '\\.\PhysicalDrive1'

VBoxManage.exe internalcommands  listpartitions -rawdisk '\\.\PhysicalDrive2'

Guest OS setup

Login to the Guest OS for the rest of this section.

Create mount-point in Guest OS, and set its group to vboxsf, like other shared folder mountpoints:

   mkdir /media/internal-storage-drive

   chgrp vboxsf /media/internal-storage-drive

Run blkid to find the UUIDS for all the drives of interest, which we will need shortly: (Credit: liquidat)

  blkid /dev/sd*

Create entry in /etc/fstab in Guest. Set options appropriately based on Filesystem type. Study this example, and then write your own version to suit your system:

##### /dev/sdb – Internal storage drive

#/dev/sdb1 – Internal SSD

UUID=4957fb72-... /media/internal-storage-drive ext4    user,group,rw              0       2

#/dev/sdb2 – Swap partition we’ll ignore for now.

##### /dev/sdc: External USB drive

# /dev/sdc1: MBR or something

# /dev/sdc1: LABEL="EFI" UUID="70D6-1701" TYPE="vfat"

#noauto, at least until we install hfsplus drivers and everything looks stable
# /dev/sdc2: UUID="aa810eb7-4048-3b0f-9ecf-8a2d6841337b" LABEL="Elements-Mac" TYPE="hfsplus"
LABEL=Elements-Mac /media/external-elements-mac  hfsplus    rw,noauto              0       2

# NOT SAFE to mount ntfs drive that is also mounted by Host OS!
#/dev/sdc3: LABEL="Elements-NT" UUID="52FB51860ACFBBED" TYPE="ntfs"
#LABEL=Elements-NT  /media/external-elements-nt  ntfs    user,group,rw              0       2

# /dev/sdc4: LABEL="Linux" UUID="59a3928e-d38f-4855-837a-31d32562fac9" TYPE="ext4"
UUID=59a3928e-...   /media/external-elements-linux  ext4    user,group,rw              0       2

# /dev/sdb5: UUID="08045a62-85bc-4028-b23e-c05670d6cca0" TYPE="swap"

Make your user a member of “vboxsf” group that can mount modify mountpoint. (vbosxf is overloaded – I’m using the Shared Folder group for Host disks)

sudo usermod –append –groups vboxsf

in gparted in Guest, remove “Boot” flag (for safety, for now)

sudo gparted

Mount!

 mount /dev/sdb1    or    mount -a

Note: If the userids on the Guest OS do not match the user ids on the hard drive, you’ll have permissions problems. If you have problems, you’ll need to re-number users on the Guest OS (consult OS documentation, /etc/passwd), or change owners of files on the hard drive (chown –R).

WARNING: Do not attempt to mount any disk/partition that is also mounted by the Host system, especially not the Host boot disk!

WARNING: Some programs, like Nautilus, will auto-detect all partitions on a hard drive, and attempt to mount them if you click on an icon. Clicking the icon for any partition mounted by the Host OS will crash the Guest OS!

References: http://www.sysprobs.com/access-physical-disk-virtualbox-desktop-virtualization-software

USB devices

I thought I needed to set this up, but I was able to access my external USB hard drive using the “raw hard drive access” method above.

Audio

Should work automatically. Audio works for me when I connect to the “head” console of the VM (non-headless mode). When I connect to the VM over ssh and run a program there, sound does not transmit back to my client.

This is a general challenge of running a program on a remote machine, not VirtualBox specific.

To Be Continued…

There is a lot more Linux setup to do, but it’s not VirtualBox specific. We’ll cover that material in another article.

Managing Home Videos

We have a Kodak “Flip”-esque video camera for caputring “video snapshots” at home.
The camera comes preloaded with ArcSoft MediaImpression for Kodak. I use MediaImpression’s Import functionality to import videos (and a few stills) from the camera to %USER_PROFILE%/Videos/YYYYMMDD.

I don’t know what to do with them next, but for now, they sit there. (From my cell phone, I upload to my Youtube account. I could do the same for video files on my desktop.)

I delete files from the mobile device every time I import. This is slightly risky, but it avoids the hassle of juggling lots of old content mixed with new content when I use the device. Also, this is a clear strategy that lets me know the status of each item I’ve captured: not yet imported, imported, or deleted.

It is very important to have a backup strategy in place before we start importing media.
MediaImpression Quirk:  If a media file is open in another application (maybe you are watching it  QuickTime?), MediaImpression will silently fail to delete the file on import, leaving you perhaps confused why one or two files remain on the camera after import. Be confused no longer!

JPEG quality in Nikon 60 camera

I recently migrated my photos from my old Mac to my new Windows desktop. The migration was a bit of a pain, in part because I had 30GB of photo files to migrate. Most of those 40GB were photos from the past two years.
Looking ahead, this pattern is unsustainable. I need a smaller photo library.

There are a few general approaches to shrinking the library:

1. Fewer photos
2. Smaller photo files.
1. Fewer megapixels
2. Tighter (JPEG) compression.

I am now employing a mix of methods. This post discusses an approach to Smaller photo files through tighter compression. Ken Rockwell has an excellent article on his excellent site, that should convince you that you do not need "RAW" or "JPEG Fine" or "JPEG Super-fine" files from your DSLR camera. I'll just add that lenses and lighting will do far more to your photo quality than (non-)compression.

All you really need to know is, as Ken says: "For Nikon cameras without the Optimal Image Quality JPG mode (D1X, D70, D50, D100) I use NORMAL JPG.

I have a Nikon D60. (Thanks, Dan and Lucy!)  The Nikon D60 offers four photo storage quality modes. I took some test photos to decode their meaning. Here is what they are called and what they mean:

Name	Quality	bits per pixel	Relative File size	Subjective quality loss
NEF (RAW)	100		200% – 400% (10 MB)
JPEG Fine	98	4	100% (2-4 MB)	None
JPEG Normal	97	2	70% (1-2 MB)	None
JPEG Basic	75	1	20% (0.4 - 0.8 MB)	Trivial
NEF (RAW) + JPEG Basic			stores two files

Conclusion: Unless you are taking photos under perfectly lit conditions with perfect focus, optical error is going to dominate picture quality, not JPEG compression.

Recommendation: Use JPEG Basic (or Normal if you are paranoid) for day-to-day shooting outside of a studio.

I will update this post with more sample photos and analysis, some time….

My Picasa Workflow

Here is how I manage my photos in Picasa.

• Import photos from camera over USB using Windows Photo Gallery –> Import
• Choose “Review, organize”
• Adjust time-slider to auto-split events appropriately.
• Open Picasa.
• Select all photos in new folder.
• Picture –> Batch Edit –> I’m Feeling Lucky
• Review photos individually.
  • Star photos I want to Sync to Web
  • Sync photos to web.
  • (My Sync settings: 1600px, which is big enough to fill most computer monitors, and saves storage space / bandwidth)

TODO:

Permanently resize photos down from 8 Megapixel “superfine” (98% JPEG ) to a more practical (smaller) file size. I don’t have a good plan for this yet.

Backup Strategy

Now that we are migrating priceless personal documents to the new machine, it’s time to talk backups!
For now, I’m using  Windows Backup and Restore.
What I back up: Files in libraries and Personal folders for all users and system image” every night at 9:00 PM.
Where I back up to: NTFS-formatted partition on an external USB 2.0 hard drive.
I think the defaults do not backup “data” files in funny location. To back up directories you’ve created like c:\data, you should add them to a Library. To do this: In Windows Explorer,  right-click on Libraries –> New –> Library, and give your new library a name, like “Data”. Then, navigate to your data directory, right-click, Include in Library, and choose a library. Now the directory will be backed-up along with the rest of the Libraries.

Migrating from iPhoto to Windows Photo Management

 

This is part of a series of posts in which I document my “switch” from Mac to Windows. This post discusses how I moved my photo libraries from Mac iPhoto to Windows.

Preface: iPhoto Library Management

I have a few challenges with iPhoto, which were part of the reason for my switch from a Mac laptop to a Windows desktop:
#1. I have too many photos to fit in an iPhoto library. I have ~20K photos, but iPhoto can only handle about ~5K before it gets sluggish and crashy.
#2. I have too many photos for my Mac laptop hard drive. I have a 120GB hard drive, which can’t hold all my software and movies and music and photos. So, I need to split off an “archive” library.
#2. I want to migrate my photos over time, while keeping my ability to process new photos without mixing up or losing anything.
iPhoto Buddy is the first part of the solution to these challenges. With iPhoto Buddy, we can split one giant iPhoto Library into several min-libraries. (iPhoto Library Manager could help too, but the free trial version didn’t work for me. If you are feeling confident and are willing to pay up front, I think it would be a good tool.)
It’s important to engage iPhoto Buddy long before your hard drive fills up, because you’ll need to temporarily make 2 copies of your iPhoto library, and you’ll want to work on your primary (fastest) storage disk.
Anyway, the general plan is to duplicate your iPhoto library, and then remove complementary chunks from each, creating two smaller libraries whose photos/albums/events combine to equal the original library. Then, we can take time to migrate the older “archive” libraries, while still adding and editing photos in the last library with our current photos. As long as we don’t edit any photos in the archive (but viewing and exporting is OK), we don’t have to worry about and data corruption/loss due to the migration.

Migration

I am running PhoShare to liberate my photos from iPhoto. I used iPhoto Buddy (described earlier) to split my library into 5 smaller libraries that are a few gigabytes in size with a few thousand photos in each. Next, run a PhoShare export that copies photos (and events, keywords, ratings, faces) from the Mac system/disk over to the Windows system/disk.
Specifically, share a directory from my Windows machine via SMB over the local network, and make that shared directory writable by my user account.
Then, instruct PhoShare export the photos to that SMB-mounted volume. Important PhoShare settings to note:

1. Export events, not albums, since we don’t want duplicates of all the files. (We have to make a choice here; there is no good way to keep Events and Albums / Smart Albums.)
2. Disable “Use file links”, since we are exporting to a remote disk.
3. Add {yyyy}-{mm}-{dd} to the “Folder names”, to help the Windows photo management tools (Windows Live Photo Gallery, or Picasa) sort the event by date.
4. Use {ascii_name} in Folder names, and {ascii title} in File names, to keep special characters from causing trouble in file names. I chose {title_description} for the captions.

Please read the rest of this article before executing the next step, which will take quite a bit of time and disk space. You need to make some decisions that will affect your choices in the PhoShare export.
Do a Dry Run first and check that the dry run log shows reasonable-looking behavior (check filenames especially). Then take a moment to be sure that you will be able to ensure the Mac and Windows machines stay networked together for the next 2-24 hours.  Finally, Start Export, and let PhoShare do its work.
Repeat this whole section for each “archive” iPhoto Library. When you are ready, you can choose to keep the Mac around for your “current” library, or choose take a break from photo management for the day or so it takes to move the “current” library to Windows and get set up there.

Managing Photos on Windows

Once a photo library is exported to Windows, feel free to start editing/re-organizing the photos on Windows. I recommend a light touch at first, getting accustomed to how Windows photo management works, and finding a workflow that you are happy with before you cut ties with the Mac. If something goes horribly wrong, you can go back the Mac and re-export the library from scratch. It’s nice if you don’t do too much throwaway work on Windows before you make your final decisions, though, so be careful at first.
Windows Live Essentials Photo Gallery and Picasa are your two main “gallery management” options. Neither one of them hijacks your files and directories like iPhoto, so feel free to mix and match them. They are both uglier than iPhoto, and both have both missing and extra features; adapt to your new world. Cheer yourself up by making nice panoramas in Photo Gallery, with much less effort than the wonderful-but-complicated Hugin required back on the Mac.
Dealing with Originals
If you are a hoarder / constructivist like me, you cannot bear the thought of losing your original (over-sized, miscolored) photos. Photo management programs (iPhoto, Picasa, Live Photo Gallery) each have their own superficially-incompatible ways of preserving orginal photos.

• iPhoto maintains two directories: Masters and Previews (aka Originals and Modified, respectively), and uses the iPhoto UI to “overlay” the Preview tree over the Masters tree, so that you see all Preview photos, and also and Masters photos that are not “shadowed” by a Previews photo with the same filename.
• Picasa (3+) maintains a “.picasaoriginals” directory inside each photo directory, and *moves* original master photos there before saving a modified photo.
• Live Photo Gallery maintains a single folder  “Originals” with a pool of all originals, and uses some database system to match modified photos to originals.

In my opinion, Picasa’s model is the safest and most correct. This may the single reason to use Picasa for all bulk management of changes, and only use Windows Photo Gallery for one-at-a-time edits of a photo after Picasa has safely stowed an original.
PhoShare follows the Picasa model. After exporting from PhoShare, your iPhoto Originals will be split into two camps: Pictures that were never modified will have their Original file be promoted (moved) to the “main” photo folder. Only photos that were already modified (and so had two versions already in iPhoto) will have an Original version in the PhoShare export.
If you want to “port” the Originals folder to Picasa, you can simply (manually, or using a script) rename all those “Originals” folders to “.picasaoriginals”. PhoShare can also do this for you, but there is no GUI control for this option. You’ll need to use the command-line interface to pass the “—picasaoriginals” option to PhoShare.

To-Do

• Split the Photo Event folders into Year/Month sub-directories, for easier navigation
• Port Star-ratings (which PhoShare moved to “tags”) from EXIF to Windows Photo Gallery format?
• Set up Picasa/Flickr sync.
• Enjoy managing photos on Windows?

Migrating from Mac to Windows

I got a new desktop PC, and I would like to make it the new “authoritative” home for all my Documents and media content. This means I need to migrate all my files from my old Mac laptop to my PC.
It is very important to consider the notion of “authoritative source” for any data (photos, bank statements, homework, love letters, etc). If we copy our files from one machine to the other, but then go on to make changes to those files on both machines, we’ll end up in a hopeless mess. So it is important to move the files:  Copy to the new machine, Archive the old machine’s files, then Delete the old machine’s files.
Later on, we can “check out” files from the new authoritative machine, to temporarily work on them on the laptop, but we must be vigilant to “check in” the changes back to the authoritative machine. (A later post will discuss tools we can use for this, including SMB Network shares, and Concurrent Version Control Systems)
OK, so here’s how I moved my files.

Setup

Our plan is to hook up both the old Mac and the new Windows machine to the same local network; Share my home directory over the network; and then, on the Mac, send (copy) files over to Windows.
There are three distinct categories of storage where the old files are kept:

1. Internal HFS+ filesystem (main System disk)
2. External HFS+ filesystem
3. External VFAT (Windows-compatible) filesystem

We’ll use an appropriate strategy for each.
There are two main categories of files we need to handle:

1. Platform-specific-structured files that need special care (iPhoto Library, mail, etc)
2. Simple cross-platform files (Office documents, text files, maybe music files, CD/DVD images) that we can just copy over

Let’s discuss the Internal HFS+ filesystem first.

Platform-specific-structured files

We need to use special tools to convert our Mac-specific files and bundles.

1. iPhoto Library: I’ve written an article dedicated just to this
2. iMovie projects: ??? For now, make an archive, and save it somewhere.

Let’s take a moment here to remind ourselves to avoid platform-specific content bundles (iPhoto Library) in the future.

Simple cross-platform files

There are a few two basic approaches:

1. Copy EVERYTHING (except System files and Applications) over, and then sit on the Windows machine and delete files we don’t want to keep. This uses a lot of disk space, but is relatively quick (one long “bulk” wait at the start, and then delete files at your leisure), and it’s also safe.
2. Copy files over one folder at a time. This is more labor intensive, and it potentially error prone (in case we forget something important), but uses less disk space on the new machine, and helps clean out old cruft.
3. Hybrid approach:
   1. Copy over some special content using a custom approach: Copy very large files (video files, music, disk images) first.
   2. Delete (or move into quarantine) these folders on the old machine.
   3. Do a little spring cleaning, and delete large files that we don’t intend to ever user again.
   4. Bulk-copy everything else, by copying high-level directories like /Users/<you> and any others you have.

External HFS+ filesystem
Windows cannot read HFS+ filesystems. Linux can read and write HFS+ filesystems. So, we’ll use Linux. Here’s our plan:

1. Run Linux (in VirtualBox)
2. Mount the external drive to the Guest OS
3. Migrate files as discussed earlier (for the main Mac system disk), choosing a Shared Folder as the destination for the files (or an SMB share, if you want), so that the files finally land on the Windows Host OS.
4. To clean up / delete the external drive, we have two choices:
   1. Reformat the whole drive (or partition), saying goodbye to HFS+
   2. Selectively delete files, and then use a partition tool to resize the partition and make room for other non-HFS content on the disk.

Steps 1 and 2 are discussed in another article in this series.
This last step (4.2)  has a twist: In order to delete (or edit) files on an HFS+ disk, we need to disable journaling (from the Mac OS) before mounting on the Linux OS.

Setting up Windows Desktop - Part 2: Configuration

A clean installation Windows in its factory-fresh state is broken. In this article we will fix a few things.
Prevent self-waking from Sleep/Hibernate
In the default configuration, Windows will wake the computer from sleep and hibernate on any network activity.

Right click on your “My Computer” then select Properties.

Click Device Manager on the left side of the Properties window.

Check your Network card on the Network Adapters (Click on the + sign to expand).

Right click on your network card and select properties.

Go to the Power Management tab and either:

• Untick the option to prevent your network card from waking up your Windows.  -OR -
• Tick the option to “Only allow a magic packet to wake the computer”.

Credit and more details about sleep management: http://www.cravingtech.com/fix-windows-vista7-sleep-mode-from-waking-up-by-itself.html
Tech Details: http://technet.microsoft.com/en-us/library/ee617165%28WS.10%29.aspx

Start –> Folder Options –> View:

• CHECK Hidden files and folders: Show hidden file, folders, and drives
• UNcheck Hide extensions for known file types  
• CHECK Restore previous folder windows at logon
• CHECK Show drive letters
• CHECK Launch folder windows in a separate process

(I’d like to show System files as well, but Windows has a bunch of bogus Junctions (NetHood, Application Data, etc) that exist for compatibility with broken legacy software, and they really clutter up the Explorer.)
Hide non-Windows partitions
If you have an hard disk partition with ext (Linux) or HFS (Mac) format, Windows will map the partition to a drive letter automatically, and continually offer to format the partition to Windows format. top protect your data and yourself, hide these drives from windows:

1. Start –> “Disk Management” (which appears incorrectly as “Create and format hard disk partitions”)
2. Right-click on the partition entry (either table cell at top, or graphical view on bottom), and choose “Change Drive Letter and Paths…” –> Remove
3. While you are here, feel free to re-map other drives/partitions to your liking. Do this as early as possible, before you start creating Libraries and shortcuts and scripts and such that refer to the old drive letters.

Credit: http://forums.techarena.in/tips-tweaks/1179239.htm

Windows Media Player Network Sharing Service has a habit of wasting CPU while doing nothing of value. It runs as a process called "wmpnetwk.exe”. Disable it: Start –> “Services” –> Windows Media Player Network Sharing Service  -> Startup Type : Manual and also click Stop. More info.

Remove some epileptic bling. Start –> “Adjust the appearance and performance of Windows”. (This is subjective, but I recommend these settings to improve the usability of Aero desktop)
UN-check:

• Animate windows when minimizing and maximizing
• Enable Aero Peek     (Enable: Alt-Tab shows previews  Disable: Alt-Tab shows icons)
• Fade or slide menus into view
• Fade or slide ToolTips into view
• Fade out menu items after clicking
• Save taskbar thumbnail previews
• Show translucent selection rectangle

Thin down the huge window borders. Start –> “Change window color and and metrics”.

• Item: Active Window Border. Size: 2
• Item: Border Padding: Size: 2

Configuring Applications

If you had to install Adobe Reader, you might need to fix its broken default DPI setting in order to make document look good. Edit –> Preferences –> Resolution –> Use System Setting.

Setting up Windows Desktop - Part 1: Software

This is the first in a series of techie articles, where I describe my recent experience setting up my Windows desktop PC for Math & Programming.

I recently purchased a new Desktop PC. This system is quite powerful (compared to my old Mac laptop), and I would like to put it to full use. Besides web browsing and other “client computing”, I would like to migrate my Mac-based projects (developing 3qual in Eclipse, Photo library management, etc) to this machine. I would also like to run a Linux system where I can explore programs and developer tools that are designed for Linux, and I want to access the Linux-formatted (ext-format) hard drive in my system, if possible.

This series outlines the solutions I have found.

To get started, let’s install useful software.

Bootstrap with Ninite

Ninite provides a great tool to build a bundle installer with many popular software packages. Visite http://ninite.com and configure a custom installer for software you like. 

Beware, though: some it is junkware. Do not blindly include everything!

I recommend the following:

General programs

1. Web Browsers: Chrome, Firefox, Safari, and/or Opera, as many as you like. If you prefer to run Chrome Beta or Firefox Beta, skip those here and download the Betas from their sources.
2. Messaging: Skype.  Trillian is also popular. (I use Gmail’s web-based Google Talk & AIM client)
3. Media:
   1. VLC for playing video and DVDs
   2. foobar2000 for playing music
   3. K-Lite Codecs, CCCP for handling various audio/video formats
4. Runtimes: Flash, Java, .NET, Silverlight, AIR, Shockwave, for running applets of various natures.
5. Imaging:
   1. Paint.NET (better than Paint)
   2. Picasa photo library manager and editor
   3. GIMP 2.6: PhotoShop alternative image editor (unless you want to get GIMP 2.7 from the source)
   4. Inkspace vector graphics editor
6. Documents:
   1. LibreOffice  alternative to Microsoft Office
   2. PDFCreator or CutePDF for Print-to-PDF (like the Mac does)
   3. SumatraPDF or Foxit to read PDFs. Adobe Reader if you must.
7. Security
   1. Essentials by Microsoft.
8. Utilities: RealVNC
9. Compression: 7-Zip for all compressed file formats.
10. Developer Tools: All of these are great.
11. Anything else you know you want, also. Beware of “Trial” versions.

As soon as you have Firefox or Chrome installed, stop using Internet Explorer.

Cygwin
If you have any interest in Unix, install Cygwin. Cygwin provides a Windows implementation of POSIX (Portable Operating System Interface for Unix). The rest of this post uses “Cygwin” and “POSIX” mostly interchangeably.

The defaults are basically fine.

Be sure to add xinit to the list of packages to install, which will provide an X Server that we will use later (when we create a Linux virtual machine).

After install, you can run bash.exe and lots of other Unix classics. If anything you like is missing, run cygwin setup.exe again and see if what you need is available for install.

Note that Cygwin creates a new home directory for you, under a directory called home that is placed under wherever you installed Cygwin. Thus,  ~ and ~username do nor point to your Windows home directory! If you prefer to have a combined Windows/POSIX home directory, move your POSIX home directory into your Windows home (“user profile”) directory:

mv $HOME $HOMEPATH\\cygwin

and then create a soft link that points from the POSIX location into your Windows home (“user profile”) directory:

ln -s $HOMEPATH\\cygwin $HOME

Special Cygwin tip: Some POSIX programs prefer Unix-style paths with forewhacks (“/”) . Most Windows programs prefer Windows-style paths with backwhacks (“\”), even when called from a Cygwin shell. To easily switch between the two forms, use cygpath, as in this example:

$ WindowsProgram.exe $(cygpath –windows ~/my_file)
$ posix_program $(cygpath –unix C:\myfile)

Programmer Tools

Programmers install these software packages:

• Git version control system: Either:
• install “git” and related packaged as part of your Cygwin installation
• Run Cygwin setup.exe and choose+install the “git” programs
• Add “export PATH=/usr/lib/git-core/:$PATH” to your ~/.profile , to make all the git commands available in your shell (especially git-gui)
• or install Git for Windows, which is a combined Cygwin+Git bundle.

Other software

If you’d like to keep an eye and finger on your system’s CPU speed, temperature, and fan speeds, install CPU-Z and SpeedFan.

For power-user system debugging, grab SysInternals Suite, featuring Process Explorer, PsKill, and more. The suite does not have an installer, so create C:\Program Files\SysInternals\ and copy all the files into there.

Many of these are command-line utilities, so “Edit environment variables for your account” and add the C:\Program Files\SysInternals\ to your PATH variable.

I’m doing a little math these days, so I installed MiKTeX TeX engine, and LyX LaTeX GUI front-end.

For more math fun, you can install RStudio to run the R statistics programming system, or you can install RStudio Server on a Linux system, and connect to that through your browser.

To keep tabs on disk space usage, WinDirStat.

To fix file attributes (like incorrect dates from cameras with mis-set clocks), SKTimeStamp.

To Be Continued…

Whew! Now we have a mostly functional Windows workstation.
There’s a lot still to do. Here’s what’s coming up later in the series:

1. Migrate our photo libraries from Mac iPhoto to Windows.
2. Installing a virtual machine inside on our PC, so we can run a Linux on Windows system, and run real Linux programs side by side with Windows programs.
3. Recurse! Set up our favorite Linux programs on our virtual Linux machine.

Welcome to Pascalcula Patterns!

Welcome to the new Pascalcula Pattern blog!

Here we will discuss techie and business topics: Personal computing, software development, web oftware, mobile software (especially Android), running a small software business, and more. The scope will grow over time, but for now, it's main focus is to contain all the geeky stuff that can't compete with the baby pictures on our personal blogs :-)

The first few posts will document my recent adventure in "Switching" from Mac to Windows, including setting up my new Windows system for photo management, software development, and of course running Linux (on a virtual machine).