Text file
talks/2014/camlistore.slide
1 Camlistore: Android, ARM, App Engine, anywhere.
2
3 25 June 2014
4 Tags: camlistore
5
6 Brad Fitzpatrick
7 Gopher, Google
8 @bradfitz
9 bradfitz@golang.org
10 http://bradfitz.com/
11 http://camlistore.org/
12
13 * Me
14
15 - bradfitz
16 - LiveJournal, memcached, OpenID, ...
17 - Google
18 - Go Standard Library, etc
19 - Camlistore
20
21 * Camlistore
22
23 * Camlistore
24
25 - "Your personal storage system for life"
26 - 4 year+ open source project:
27 .link http://camlistore.org
28 - All in Go.
29 - Early user of Go's standard library. Helped shape it.
30
31 * Camlistore
32
33 - stores, indexes, searches, syncs, shares content
34 - can optionally model POSIX filesystems
35 - lightweight object storage model, stores all mutations
36
37 Stores on:
38
39 - local disk, S3, Google Cloud Storage
40 - MySQL, Postgres, SQLite, LevelDB,
41 - MongoDB, App Engine,
42 - itself,
43 - encrypted, sharded, replicated, load balanced,
44 - ...
45
46 * Camlistore screenshots
47
48 * Web UI
49
50 .image camlistore/cam-mix-types.png _ 1000
51
52 * Location search
53
54 .image camlistore/cam-moscow.png 580 _
55
56 * Panos
57
58 .image camlistore/cam-pano.png _ 1000
59
60 * Paris + Portrait
61
62 .image camlistore/cam-paris-portrait.png _ 1000
63
64 * Non-images
65
66 .image camlistore/cam-other.png _ 1000
67
68 * FUSE
69
70 .image camlistore/cam-fuse.png _ 1000
71
72 * Android
73
74 .image camlistore/cam-android.png 600 _
75
76 * Camlistore Demo
77
78 * Camlistore does much, in many places
79
80 * Camlistore
81
82 does:
83
84 - a web & API server
85 - a FUSE filesystem (OS X, Linux)
86 - command-line tools
87
88 on:
89
90 - desktops, personal servers
91 - Raspberry Pi, etc (ARM)
92 - Android
93 - App Engine
94 - Cloud (EC2, GCE)
95
96 * Desktops, personal servers
97
98 - Linux, FreeBSD, Mac, Windows, ...
99
100 $ go {get,build,install,test} camlistore.org/{cmd,pkg,server}/...
101
102 - Occasional use of +build windows, etc.
103 - Nothing fancy.
104
105 * Raspberry Pi
106
107 - Most ARM devices are very slow
108 - Cross compile from any OS, architecture on a fast machine:
109
110 $ GOARCH=arm GOOS=linux go build camlistore.org/server/camlistored
111
112 - ... then copy binary to the slow ARM device.
113 - Popular helper tool:
114
115 .link https://github.com/davecheney/golang-crosscompile
116
117 - For debugging: ChromeOS + crouton is nice, pretty fast
118
119 * App Engine
120
121 - Just upload the code.
122 - App Engine APIs require context plumbing.
123 - Ugly hacks avoid context plumbing, but don't.
124 .link https://developers.google.com/appengine/docs/go/reference#Context
125 .link https://pkg.go.dev/camlistore.org/pkg/context
126 .link https://pkg.go.dev/code.google.com/p/go.net/context
127 - Restructure code to not assume long-running goroutines; handlers only
128 - No unsafe, syscall, etc: use build tags on files:
129
130 // +build !appengine
131
132 * Cloud
133
134 - Same as desktop.
135 - Static binaries make deployment easy
136 - Even easier:
137 - Embed resources (CSS, JS, HTML, images) with fileembed
138 - Camlistore has: camlistore.org/pkg/fileembed + genfileembed
139
140 * fileembed
141
142 Definition:
143
144 package fileembed
145 func (f *Files) Open(filename string) (http.File, error)
146
147 Use in Camlistore:
148
149 package server
150 uistatic.Files = &fileembed.Files{
151 DirFallback: ui.uiDir,
152 Listable: true,
153 // In dev_appserver, allow edit-and-reload without
154 // restarting. In production, though, it's faster to just
155 // slurp it in.
156 SlurpToMemory: uistatic.IsProdAppEngine,
157 }
158
159 - Just reload in dev; embedded in binary in production.
160
161 * Camlistore's genfileembed
162
163 $ genfileembed ./path/to/dir
164
165 - generates zindex.html.go
166
167 * Embedding resources
168
169 - Many options in the Go community
170 - Coming to Go 1.4 built-in
171
172 * Web server
173
174 - Obvious
175 - net/http is great: no need for Apache, nginx, etc
176 - WebSockets support is great and easy
177
178 * FUSE
179
180 - Native filesystem on OS X & Linux
181 - hanwen/gofuse (then only Linux) -> rsc/fuse (then only Mac) -> Camlistore fork (both) -> bazil.org (community maintained)
182 - Easy to write fast filesystems in Go
183
184 * Android
185
186 - child process
187 - on Application start, unzip arm binary to Unix (not media) filesystem
188 .link https://camlistore.googlesource.com/camlistore/+/master/clients/android/src/org/camlistore/UploadApplication.java UploadApplication.java
189 .link https://camlistore.googlesource.com/camlistore/+/master/clients/android/src/org/camlistore/UploadThread.java UploadThread.java
190 - build with Docker (demo)
191 .link https://camlistore.googlesource.com/camlistore/+/master/clients/android/devenv/Dockerfile Dockerfile
192 .link https://camlistore.googlesource.com/camlistore/+/master/clients/android/Makefile Makefile
193 .link https://camlistore.googlesource.com/camlistore/+/master/clients/android/build-in-docker.pl build-in-docker.pl
194
195 * Command-line tools
196
197 $ cmdtool [--global=foo] subcmd [--subcommand opts]
198
199 .link http://camlistore.org/pkg/cmdmain/
200 .link https://camlistore.googlesource.com/camlistore/+/master/cmd/camtool/describe.go camtool's describe.go
201 .link https://github.com/codegangsta/cli Popular: codegangsta/cli
202
203 * In summary
204
205 Go is awesome and runs lots of places, easily.
206
207 - Web, CLI, FUSE, x86, ARM, Android, App Engine
208
View as plain text