Google Git
Sign in
chromium / external / gitster / git-htmldocs / d330432c3820c3699d45f457895f05ef6cbcec91 / . / git-fast-import.html
blob: 0964fa1c67c0f3d3764cb1515466565116820f79 [file] [log] [blame]
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
2 "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
3<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
4<head>
5<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]6<meta name="generator" content="AsciiDoc 8.2.5" />
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]7<style type="text/css">
8/* Debug borders */
9p, li, dt, dd, div, pre, h1, h2, h3, h4, h5, h6 {
10/*
11 border: 1px solid red;
12*/
13}
14
15body {
16 margin: 1em 5% 1em 5%;
17}
18
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]19a {
20 color: blue;
21 text-decoration: underline;
22}
23a:visited {
24 color: fuchsia;
25}
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]26
27em {
28 font-style: italic;
29}
30
31strong {
32 font-weight: bold;
33}
34
35tt {
36 color: navy;
37}
38
39h1, h2, h3, h4, h5, h6 {
40 color: #527bbd;
41 font-family: sans-serif;
42 margin-top: 1.2em;
43 margin-bottom: 0.5em;
44 line-height: 1.3;
45}
46
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]47h1, h2, h3 {
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]48 border-bottom: 2px solid silver;
49}
50h2 {
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]51 padding-top: 0.5em;
52}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]53h3 {
54 float: left;
55}
56h3 + * {
57 clear: left;
58}
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]59
60div.sectionbody {
61 font-family: serif;
62 margin-left: 0;
63}
64
65hr {
66 border: 1px solid silver;
67}
68
69p {
70 margin-top: 0.5em;
71 margin-bottom: 0.5em;
72}
73
74pre {
75 padding: 0;
76 margin: 0;
77}
78
79span#author {
80 color: #527bbd;
81 font-family: sans-serif;
82 font-weight: bold;
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]83 font-size: 1.1em;
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]84}
85span#email {
86}
87span#revision {
88 font-family: sans-serif;
89}
90
91div#footer {
92 font-family: sans-serif;
93 font-size: small;
94 border-top: 2px solid silver;
95 padding-top: 0.5em;
96 margin-top: 4.0em;
97}
98div#footer-text {
99 float: left;
100 padding-bottom: 0.5em;
101}
102div#footer-badges {
103 float: right;
104 padding-bottom: 0.5em;
105}
106
107div#preamble,
108div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
109div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
110div.admonitionblock {
111 margin-right: 10%;
112 margin-top: 1.5em;
113 margin-bottom: 1.5em;
114}
115div.admonitionblock {
116 margin-top: 2.5em;
117 margin-bottom: 2.5em;
118}
119
120div.content { /* Block element content. */
121 padding: 0;
122}
123
124/* Block element titles. */
125div.title, caption.title {
126 font-family: sans-serif;
127 font-weight: bold;
128 text-align: left;
129 margin-top: 1.0em;
130 margin-bottom: 0.5em;
131}
132div.title + * {
133 margin-top: 0;
134}
135
136td div.title:first-child {
137 margin-top: 0.0em;
138}
139div.content div.title:first-child {
140 margin-top: 0.0em;
141}
142div.content + div.title {
143 margin-top: 0.0em;
144}
145
146div.sidebarblock > div.content {
147 background: #ffffee;
148 border: 1px solid silver;
149 padding: 0.5em;
150}
151
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]152div.listingblock {
153 margin-right: 0%;
154}
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]155div.listingblock > div.content {
156 border: 1px solid silver;
157 background: #f4f4f4;
158 padding: 0.5em;
159}
160
161div.quoteblock > div.content {
162 padding-left: 2.0em;
163}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]164
165div.attribution {
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]166 text-align: right;
167}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]168div.verseblock + div.attribution {
169 text-align: left;
170}
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]171
172div.admonitionblock .icon {
173 vertical-align: top;
174 font-size: 1.1em;
175 font-weight: bold;
176 text-decoration: underline;
177 color: #527bbd;
178 padding-right: 0.5em;
179}
180div.admonitionblock td.content {
181 padding-left: 0.5em;
182 border-left: 2px solid silver;
183}
184
185div.exampleblock > div.content {
186 border-left: 2px solid silver;
187 padding: 0.5em;
188}
189
190div.verseblock div.content {
191 white-space: pre;
192}
193
194div.imageblock div.content { padding-left: 0; }
195div.imageblock img { border: 1px solid silver; }
196span.image img { border-style: none; }
197
198dl {
199 margin-top: 0.8em;
200 margin-bottom: 0.8em;
201}
202dt {
203 margin-top: 0.5em;
204 margin-bottom: 0;
205 font-style: italic;
206}
207dd > *:first-child {
208 margin-top: 0;
209}
210
211ul, ol {
212 list-style-position: outside;
213}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]214div.olist2 ol {
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]215 list-style-type: lower-alpha;
216}
217
218div.tableblock > table {
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]219 border: 3px solid #527bbd;
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]220}
221thead {
222 font-family: sans-serif;
223 font-weight: bold;
224}
225tfoot {
226 font-weight: bold;
227}
228
229div.hlist {
230 margin-top: 0.8em;
231 margin-bottom: 0.8em;
232}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]233div.hlist td {
234 padding-bottom: 5px;
235}
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]236td.hlist1 {
237 vertical-align: top;
238 font-style: italic;
239 padding-right: 0.8em;
240}
241td.hlist2 {
242 vertical-align: top;
243}
244
245@media print {
246 div#footer-badges { display: none; }
247}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]248
249div#toctitle {
250 color: #527bbd;
251 font-family: sans-serif;
252 font-size: 1.1em;
253 font-weight: bold;
254 margin-top: 1.0em;
255 margin-bottom: 0.1em;
256}
257
258div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
259 margin-top: 0;
260 margin-bottom: 0;
261}
262div.toclevel2 {
263 margin-left: 2em;
264 font-size: 0.9em;
265}
266div.toclevel3 {
267 margin-left: 4em;
268 font-size: 0.9em;
269}
270div.toclevel4 {
271 margin-left: 6em;
272 font-size: 0.9em;
273}
274include1::./stylesheets/xhtml11-manpage.css[]
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]275/* Workarounds for IE6's broken and incomplete CSS2. */
276
277div.sidebar-content {
278 background: #ffffee;
279 border: 1px solid silver;
280 padding: 0.5em;
281}
282div.sidebar-title, div.image-title {
283 font-family: sans-serif;
284 font-weight: bold;
285 margin-top: 0.0em;
286 margin-bottom: 0.5em;
287}
288
289div.listingblock div.content {
290 border: 1px solid silver;
291 background: #f4f4f4;
292 padding: 0.5em;
293}
294
295div.quoteblock-content {
296 padding-left: 2.0em;
297}
298
299div.exampleblock-content {
300 border-left: 2px solid silver;
301 padding-left: 0.5em;
302}
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]303
304/* IE6 sets dynamically generated links as visited. */
305div#toc a:visited { color: blue; }
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]306</style>
307<title>git-fast-import(1)</title>
308</head>
309<body>
310<div id="header">
311<h1>
312git-fast-import(1) Manual Page
313</h1>
314<h2>NAME</h2>
315<div class="sectionbody">
316<p>git-fast-import -
Junio C Hamano053827f2007-02-14 07:23:58[diff] [blame]317 Backend for fast Git data importers
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]318</p>
319</div>
320</div>
321<h2>SYNOPSIS</h2>
322<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]323<div class="para"><p>frontend | <em>git fast-import</em> [options]</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]324</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]325<h2 id="_description">DESCRIPTION</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]326<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]327<div class="para"><p>This program is usually not what the end user wants to run directly.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]328Most end users want to use one of the existing frontend programs,
329which parses a specific type of foreign source and feeds the contents
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]330stored there to <em>git fast-import</em>.</p></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]331<div class="para"><p>fast-import reads a mixed command/data stream from standard input and
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]332writes one or more packfiles directly into the current repository.
333When EOF is received on standard input, fast import writes out
334updated branch and tag refs, fully updating the current repository
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]335with the newly imported data.</p></div>
336<div class="para"><p>The fast-import backend itself can import into an empty repository (one that
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]337has already been initialized by <em>git init</em>) or incrementally
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]338update an existing populated repository. Whether or not incremental
339imports are supported from a particular foreign source depends on
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]340the frontend program in use.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]341</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]342<h2 id="_options">OPTIONS</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]343<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]344<div class="vlist"><dl>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]345<dt>
346--date-format=&lt;fmt&gt;
347</dt>
348<dd>
349<p>
350 Specify the type of dates the frontend will supply to
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]351 fast-import within <tt>author</tt>, <tt>committer</tt> and <tt>tagger</tt> commands.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]352 See &#8220;Date Formats&#8221; below for details about which formats
353 are supported, and their syntax.
354</p>
355</dd>
356<dt>
357--force
358</dt>
359<dd>
360<p>
361 Force updating modified existing branches, even if doing
362 so would cause commits to be lost (as the new commit does
363 not contain the old commit).
364</p>
365</dd>
366<dt>
367--max-pack-size=&lt;n&gt;
368</dt>
369<dd>
370<p>
Junio C Hamanod3304322010-02-05 03:28:27[diff] [blame^]371 Maximum size of each output packfile.
372 The default is 4 GiB as that is the maximum allowed
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]373 packfile size (due to file format limitations). Some
374 importers may wish to lower this, such as to ensure the
375 resulting packfiles fit on CDs.
376</p>
377</dd>
378<dt>
Junio C Hamano5c246f22010-02-03 07:34:53[diff] [blame]379--big-file-threshold=&lt;n&gt;
380</dt>
381<dd>
382<p>
383 Maximum size of a blob that fast-import will attempt to
384 create a delta for, expressed in bytes. The default is 512m
385 (512 MiB). Some importers may wish to lower this on systems
386 with constrained memory.
387</p>
388</dd>
389<dt>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]390--depth=&lt;n&gt;
391</dt>
392<dd>
393<p>
394 Maximum delta depth, for blob and tree deltification.
395 Default is 10.
396</p>
397</dd>
398<dt>
399--active-branches=&lt;n&gt;
400</dt>
401<dd>
402<p>
403 Maximum number of branches to maintain active at once.
404 See &#8220;Memory Utilization&#8221; below for details. Default is 5.
405</p>
406</dd>
407<dt>
408--export-marks=&lt;file&gt;
409</dt>
410<dd>
411<p>
412 Dumps the internal marks table to &lt;file&gt; when complete.
413 Marks are written one per line as <tt>:markid SHA-1</tt>.
414 Frontends can use this file to validate imports after they
Junio C Hamanod15328a2007-03-09 09:06:40[diff] [blame]415 have been completed, or to save the marks table across
416 incremental runs. As &lt;file&gt; is only opened and truncated
417 at checkpoint (or completion) the same path can also be
418 safely given to --import-marks.
419</p>
420</dd>
421<dt>
422--import-marks=&lt;file&gt;
423</dt>
424<dd>
425<p>
426 Before processing any input, load the marks specified in
427 &lt;file&gt;. The input file must exist, must be readable, and
428 must use the same format as produced by --export-marks.
429 Multiple options may be supplied to import more than one
430 set of marks. If a mark is defined to different values,
431 the last file wins.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]432</p>
433</dd>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]434<dt>
Junio C Hamano6ce6b6c2010-01-18 01:25:50[diff] [blame]435--relative-marks
436</dt>
437<dd>
438<p>
439 After specifying --relative-marks= the paths specified
440 with --import-marks= and --export-marks= are relative
441 to an internal directory in the current repository.
442 In git-fast-import this means that the paths are relative
443 to the .git/info/fast-import directory. However, other
444 importers may use a different location.
445</p>
446</dd>
447<dt>
448--no-relative-marks
449</dt>
450<dd>
451<p>
452 Negates a previous --relative-marks. Allows for combining
453 relative and non-relative marks by interweaving
454 --(no-)-relative-marks= with the --(import|export)-marks=
455 options.
456</p>
457</dd>
458<dt>
Junio C Hamano9dd8bb02007-02-12 07:15:35[diff] [blame]459--export-pack-edges=&lt;file&gt;
460</dt>
461<dd>
462<p>
463 After creating a packfile, print a line of data to
464 &lt;file&gt; listing the filename of the packfile and the last
465 commit on each branch that was written to that packfile.
466 This information may be useful after importing projects
467 whose total object set exceeds the 4 GiB packfile limit,
468 as these commits can be used as edge points during calls
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]469 to <em>git pack-objects</em>.
Junio C Hamano9dd8bb02007-02-12 07:15:35[diff] [blame]470</p>
471</dd>
472<dt>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]473--quiet
474</dt>
475<dd>
476<p>
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]477 Disable all non-fatal output, making fast-import silent when it
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]478 is successful. This option disables the output shown by
479 --stats.
480</p>
481</dd>
482<dt>
483--stats
484</dt>
485<dd>
486<p>
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]487 Display some basic statistics about the objects fast-import has
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]488 created, the packfiles they were stored into, and the
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]489 memory used by fast-import during this run. Showing this output
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]490 is currently the default, but can be disabled with --quiet.
491</p>
492</dd>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]493</dl></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]494</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]495<h2 id="_performance">Performance</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]496<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]497<div class="para"><p>The design of fast-import allows it to import large projects in a minimum
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]498amount of memory usage and processing time. Assuming the frontend
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]499is able to keep up with fast-import and feed it a constant stream of data,
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]500import times for projects holding 10+ years of history and containing
501100,000+ individual commits are generally completed in just 1-2
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]502hours on quite modest (~$2,000 USD) hardware.</p></div>
503<div class="para"><p>Most bottlenecks appear to be in foreign source data access (the
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]504source just cannot extract revisions fast enough) or disk IO (fast-import
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]505writes as fast as the disk will take the data). Imports will run
506faster if the source data is stored on a different drive than the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]507destination Git repository (due to less IO contention).</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]508</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]509<h2 id="_development_cost">Development Cost</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]510<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]511<div class="para"><p>A typical frontend for fast-import tends to weigh in at approximately 200
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]512lines of Perl/Python/Ruby code. Most developers have been able to
513create working importers in just a couple of hours, even though it
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]514is their first exposure to fast-import, and sometimes even to Git. This is
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]515an ideal situation, given that most conversion tools are throw-away
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]516(use once, and never look back).</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]517</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]518<h2 id="_parallel_operation">Parallel Operation</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]519<div class="sectionbody">
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]520<div class="para"><p>Like <em>git push</em> or <em>git fetch</em>, imports handled by fast-import are safe to
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]521run alongside parallel <tt>git repack -a -d</tt> or <tt>git gc</tt> invocations,
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]522or any other Git operation (including <em>git prune</em>, as loose objects
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]523are never used by fast-import).</p></div>
524<div class="para"><p>fast-import does not lock the branch or tag refs it is actively importing.
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]525After the import, during its ref update phase, fast-import tests each
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]526existing branch ref to verify the update will be a fast-forward
527update (the commit stored in the ref is contained in the new
528history of the commit to be written). If the update is not a
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]529fast-forward update, fast-import will skip updating that ref and instead
530prints a warning message. fast-import will always attempt to update all
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]531branch refs, and does not stop on the first failure.</p></div>
Junio C Hamano167b1382010-01-31 23:04:31[diff] [blame]532<div class="para"><p>Branch updates can be forced with --force, but it's recommended that
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]533this only be used on an otherwise quiet repository. Using --force
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]534is not necessary for an initial import into an empty repository.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]535</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]536<h2 id="_technical_discussion">Technical Discussion</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]537<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]538<div class="para"><p>fast-import tracks a set of branches in memory. Any branch can be created
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]539or modified at any point during the import process by sending a
540<tt>commit</tt> command on the input stream. This design allows a frontend
541program to process an unlimited number of branches simultaneously,
542generating commits in the order they are available from the source
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]543data. It also simplifies the frontend programs considerably.</p></div>
544<div class="para"><p>fast-import does not use or alter the current working directory, or any
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]545file within it. (It does however update the current Git repository,
546as referenced by <tt>GIT_DIR</tt>.) Therefore an import frontend may use
547the working directory for its own purposes, such as extracting file
548revisions from the foreign source. This ignorance of the working
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]549directory also allows fast-import to run very quickly, as it does not
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]550need to perform any costly file update operations when switching
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]551between branches.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]552</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]553<h2 id="_input_format">Input Format</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]554<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]555<div class="para"><p>With the exception of raw file data (which Git does not interpret)
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]556the fast-import input format is text (ASCII) based. This text based
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]557format simplifies development and debugging of frontend programs,
558especially when a higher level language such as Perl, Python or
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]559Ruby is being used.</p></div>
560<div class="para"><p>fast-import is very strict about its input. Where we say SP below we mean
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]561<strong>exactly</strong> one space. Likewise LF means one (and only one) linefeed.
562Supplying additional whitespace characters will cause unexpected
563results, such as branch names or file names with leading or trailing
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]564spaces in their name, or early termination of fast-import when it encounters
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]565unexpected input.</p></div>
566<h3 id="_stream_comments">Stream Comments</h3><div style="clear:left"></div>
567<div class="para"><p>To aid in debugging frontends fast-import ignores any line that
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]568begins with <tt>#</tt> (ASCII pound/hash) up to and including the line
569ending <tt>LF</tt>. A comment line may contain any sequence of bytes
570that does not contain an LF and therefore may be used to include
571any detailed debugging information that might be specific to the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]572frontend and useful when inspecting a fast-import data stream.</p></div>
573<h3 id="_date_formats">Date Formats</h3><div style="clear:left"></div>
574<div class="para"><p>The following date formats are supported. A frontend should select
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]575the format it will use for this import by passing the format name
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]576in the --date-format=&lt;fmt&gt; command line option.</p></div>
577<div class="vlist"><dl>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]578<dt>
579<tt>raw</tt>
580</dt>
581<dd>
582<p>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]583 This is the Git native format and is <tt>&lt;time&gt; SP &lt;offutc&gt;</tt>.
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]584 It is also fast-import's default format, if --date-format was
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]585 not specified.
586</p>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]587<div class="para"><p>The time of the event is specified by <tt>&lt;time&gt;</tt> as the number of
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]588seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]589written as an ASCII decimal integer.</p></div>
590<div class="para"><p>The local offset is specified by <tt>&lt;offutc&gt;</tt> as a positive or negative
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]591offset from UTC. For example EST (which is 5 hours behind UTC)
592would be expressed in <tt>&lt;tz&gt;</tt> by &#8220;-0500&#8221; while UTC is &#8220;+0000&#8221;.
593The local offset does not affect <tt>&lt;time&gt;</tt>; it is used only as an
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]594advisement to help formatting routines display the timestamp.</p></div>
595<div class="para"><p>If the local offset is not available in the source material, use
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]596&#8220;+0000&#8221;, or the most common local offset. For example many
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]597organizations have a CVS repository which has only ever been accessed
598by users who are located in the same location and timezone. In this
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]599case a reasonable offset from UTC could be assumed.</p></div>
600<div class="para"><p>Unlike the <tt>rfc2822</tt> format, this format is very strict. Any
601variation in formatting will cause fast-import to reject the value.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]602</dd>
603<dt>
604<tt>rfc2822</tt>
605</dt>
606<dd>
607<p>
608 This is the standard email format as described by RFC 2822.
609</p>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]610<div class="para"><p>An example value is &#8220;Tue Feb 6 11:22:18 2007 -0500&#8221;. The Git
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]611parser is accurate, but a little on the lenient side. It is the
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]612same parser used by <em>git am</em> when applying patches
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]613received from email.</p></div>
614<div class="para"><p>Some malformed strings may be accepted as valid dates. In some of
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]615these cases Git will still be able to obtain the correct date from
616the malformed string. There are also some types of malformed
617strings which Git will parse wrong, and yet consider valid.
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]618Seriously malformed strings will be rejected.</p></div>
619<div class="para"><p>Unlike the <tt>raw</tt> format above, the timezone/UTC offset information
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]620contained in an RFC 2822 date string is used to adjust the date
621value to UTC prior to storage. Therefore it is important that
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]622this information be as accurate as possible.</p></div>
623<div class="para"><p>If the source material uses RFC 2822 style dates,
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]624the frontend should let fast-import handle the parsing and conversion
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]625(rather than attempting to do it itself) as the Git parser has
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]626been well tested in the wild.</p></div>
627<div class="para"><p>Frontends should prefer the <tt>raw</tt> format if the source material
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]628already uses UNIX-epoch format, can be coaxed to give dates in that
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]629format, or its format is easily convertible to it, as there is no
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]630ambiguity in parsing.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]631</dd>
632<dt>
633<tt>now</tt>
634</dt>
635<dd>
636<p>
637 Always use the current time and timezone. The literal
638 <tt>now</tt> must always be supplied for <tt>&lt;when&gt;</tt>.
639</p>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]640<div class="para"><p>This is a toy format. The current time and timezone of this system
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]641is always copied into the identity string at the time it is being
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]642created by fast-import. There is no way to specify a different time or
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]643timezone.</p></div>
Junio C Hamano167b1382010-01-31 23:04:31[diff] [blame]644<div class="para"><p>This particular format is supplied as it's short to implement and
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]645may be useful to a process that wants to create a new commit
646right now, without needing to use a working directory or
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]647<em>git update-index</em>.</p></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]648<div class="para"><p>If separate <tt>author</tt> and <tt>committer</tt> commands are used in a <tt>commit</tt>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]649the timestamps may not match, as the system clock will be polled
650twice (once for each command). The only way to ensure that both
651author and committer identity information has the same timestamp
652is to omit <tt>author</tt> (thus copying from <tt>committer</tt>) or to use a
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]653date format other than <tt>now</tt>.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]654</dd>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]655</dl></div>
656<h3 id="_commands">Commands</h3><div style="clear:left"></div>
657<div class="para"><p>fast-import accepts several commands to update the current repository
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]658and control the current import process. More detailed discussion
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]659(with examples) of each command follows later.</p></div>
660<div class="vlist"><dl>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]661<dt>
662<tt>commit</tt>
663</dt>
664<dd>
665<p>
666 Creates a new branch or updates an existing branch by
667 creating a new commit and updating the branch to point at
668 the newly created commit.
669</p>
670</dd>
671<dt>
672<tt>tag</tt>
673</dt>
674<dd>
675<p>
676 Creates an annotated tag object from an existing commit or
677 branch. Lightweight tags are not supported by this command,
678 as they are not recommended for recording meaningful points
679 in time.
680</p>
681</dd>
682<dt>
683<tt>reset</tt>
684</dt>
685<dd>
686<p>
687 Reset an existing branch (or a new branch) to a specific
688 revision. This command must be used to change a branch to
689 a specific revision without making a commit on it.
690</p>
691</dd>
692<dt>
693<tt>blob</tt>
694</dt>
695<dd>
696<p>
697 Convert raw file data into a blob, for future use in a
698 <tt>commit</tt> command. This command is optional and is not
699 needed to perform an import.
700</p>
701</dd>
702<dt>
703<tt>checkpoint</tt>
704</dt>
705<dd>
706<p>
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]707 Forces fast-import to close the current packfile, generate its
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]708 unique SHA-1 checksum and index, and start a new packfile.
709 This command is optional and is not needed to perform
710 an import.
711</p>
712</dd>
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]713<dt>
714<tt>progress</tt>
715</dt>
716<dd>
717<p>
718 Causes fast-import to echo the entire line to its own
719 standard output. This command is optional and is not needed
720 to perform an import.
721</p>
722</dd>
Junio C Hamano6ce6b6c2010-01-18 01:25:50[diff] [blame]723<dt>
724<tt>feature</tt>
725</dt>
726<dd>
727<p>
728 Require that fast-import supports the specified feature, or
729 abort if it does not.
730</p>
731</dd>
732<dt>
733<tt>option</tt>
734</dt>
735<dd>
736<p>
737 Specify any of the options listed under OPTIONS that do not
738 change stream semantic to suit the frontend's needs. This
739 command is optional and is not needed to perform an import.
740</p>
741</dd>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]742</dl></div>
743<h3 id="_tt_commit_tt"><tt>commit</tt></h3><div style="clear:left"></div>
744<div class="para"><p>Create or update a branch with a new commit, recording one logical
745change to the project.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]746<div class="literalblock">
747<div class="content">
748<pre><tt> 'commit' SP &lt;ref&gt; LF
749 mark?
Junio C Hamano6d325df2010-01-01 00:02:47[diff] [blame]750 ('author' (SP &lt;name&gt;)? SP LT &lt;email&gt; GT SP &lt;when&gt; LF)?
751 'committer' (SP &lt;name&gt;)? SP LT &lt;email&gt; GT SP &lt;when&gt; LF
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]752 data
753 ('from' SP &lt;committish&gt; LF)?
754 ('merge' SP &lt;committish&gt; LF)?
Junio C Hamano3b70d3c2009-11-21 17:37:37[diff] [blame]755 (filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)*
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]756 LF?</tt></pre>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]757</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]758<div class="para"><p>where <tt>&lt;ref&gt;</tt> is the name of the branch to make the commit on.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]759Typically branch names are prefixed with <tt>refs/heads/</tt> in
760Git, so importing the CVS branch symbol <tt>RELENG-1_0</tt> would use
761<tt>refs/heads/RELENG-1_0</tt> for the value of <tt>&lt;ref&gt;</tt>. The value of
762<tt>&lt;ref&gt;</tt> must be a valid refname in Git. As <tt>LF</tt> is not valid in
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]763a Git refname, no quoting or escaping syntax is supported here.</p></div>
764<div class="para"><p>A <tt>mark</tt> command may optionally appear, requesting fast-import to save a
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]765reference to the newly created commit for future use by the frontend
766(see below for format). It is very common for frontends to mark
767every commit they create, thereby allowing future branch creation
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]768from any imported commit.</p></div>
769<div class="para"><p>The <tt>data</tt> command following <tt>committer</tt> must supply the commit
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]770message (see below for <tt>data</tt> command syntax). To import an empty
771commit message use a 0 length data. Commit messages are free-form
772and are not interpreted by Git. Currently they must be encoded in
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]773UTF-8, as fast-import does not permit other encodings to be specified.</p></div>
Junio C Hamano3b70d3c2009-11-21 17:37:37[diff] [blame]774<div class="para"><p>Zero or more <tt>filemodify</tt>, <tt>filedelete</tt>, <tt>filecopy</tt>, <tt>filerename</tt>,
775<tt>filedeleteall</tt> and <tt>notemodify</tt> commands
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]776may be included to update the contents of the branch prior to
777creating the commit. These commands may be supplied in any order.
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]778However it is recommended that a <tt>filedeleteall</tt> command precede
Junio C Hamano3b70d3c2009-11-21 17:37:37[diff] [blame]779all <tt>filemodify</tt>, <tt>filecopy</tt>, <tt>filerename</tt> and <tt>notemodify</tt> commands in
780the same commit, as <tt>filedeleteall</tt> wipes the branch clean (see below).</p></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]781<div class="para"><p>The <tt>LF</tt> after the command is optional (it used to be required).</p></div>
782<h4 id="_tt_author_tt"><tt>author</tt></h4>
783<div class="para"><p>An <tt>author</tt> command may optionally appear, if the author information
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]784might differ from the committer information. If <tt>author</tt> is omitted
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]785then fast-import will automatically use the committer's information for
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]786the author portion of the commit. See below for a description of
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]787the fields in <tt>author</tt>, as they are identical to <tt>committer</tt>.</p></div>
788<h4 id="_tt_committer_tt"><tt>committer</tt></h4>
789<div class="para"><p>The <tt>committer</tt> command indicates who made this commit, and when
790they made it.</p></div>
791<div class="para"><p>Here <tt>&lt;name&gt;</tt> is the person's display name (for example
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]792&#8220;Com M Itter&#8221;) and <tt>&lt;email&gt;</tt> is the person's email address
793(&#8220;cm@example.com&#8221;). <tt>LT</tt> and <tt>GT</tt> are the literal less-than (\x3c)
794and greater-than (\x3e) symbols. These are required to delimit
795the email address from the other fields in the line. Note that
796<tt>&lt;name&gt;</tt> is free-form and may contain any sequence of bytes, except
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]797<tt>LT</tt> and <tt>LF</tt>. It is typically UTF-8 encoded.</p></div>
798<div class="para"><p>The time of the change is specified by <tt>&lt;when&gt;</tt> using the date format
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]799that was selected by the --date-format=&lt;fmt&gt; command line option.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]800See &#8220;Date Formats&#8221; above for the set of supported formats, and
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]801their syntax.</p></div>
802<h4 id="_tt_from_tt"><tt>from</tt></h4>
803<div class="para"><p>The <tt>from</tt> command is used to specify the commit to initialize
Junio C Hamano5dad0832007-02-13 05:16:23[diff] [blame]804this branch from. This revision will be the first ancestor of the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]805new commit.</p></div>
806<div class="para"><p>Omitting the <tt>from</tt> command in the first commit of a new branch
Junio C Hamano5dad0832007-02-13 05:16:23[diff] [blame]807will cause fast-import to create that commit with no ancestor. This
808tends to be desired only for the initial commit of a project.
Junio C Hamano25c0b702008-03-23 09:41:37[diff] [blame]809If the frontend creates all files from scratch when making a new
810branch, a <tt>merge</tt> command may be used instead of <tt>from</tt> to start
811the commit with an empty tree.
Junio C Hamano5dad0832007-02-13 05:16:23[diff] [blame]812Omitting the <tt>from</tt> command on existing branches is usually desired,
813as the current commit on that branch is automatically assumed to
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]814be the first ancestor of the new commit.</p></div>
815<div class="para"><p>As <tt>LF</tt> is not valid in a Git refname or SHA-1 expression, no
816quoting or escaping syntax is supported within <tt>&lt;committish&gt;</tt>.</p></div>
817<div class="para"><p>Here <tt>&lt;committish&gt;</tt> is any of the following:</p></div>
818<div class="ilist"><ul>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]819<li>
820<p>
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]821The name of an existing branch already in fast-import's internal branch
Junio C Hamano167b1382010-01-31 23:04:31[diff] [blame]822 table. If fast-import doesn't know the name, it's treated as a SHA-1
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]823 expression.
824</p>
825</li>
826<li>
827<p>
828A mark reference, <tt>:&lt;idnum&gt;</tt>, where <tt>&lt;idnum&gt;</tt> is the mark number.
829</p>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]830<div class="para"><p>The reason fast-import uses <tt>:</tt> to denote a mark reference is this character
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]831is not legal in a Git branch name. The leading <tt>:</tt> makes it easy
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]832to distinguish between the mark 42 (<tt>:42</tt>) and the branch 42 (<tt>42</tt>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]833or <tt>refs/heads/42</tt>), or an abbreviated SHA-1 which happened to
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]834consist only of base-10 digits.</p></div>
835<div class="para"><p>Marks must be declared (via <tt>mark</tt>) before they can be used.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]836</li>
837<li>
838<p>
839A complete 40 byte or abbreviated commit SHA-1 in hex.
840</p>
841</li>
842<li>
843<p>
844Any valid Git SHA-1 expression that resolves to a commit. See
845 &#8220;SPECIFYING REVISIONS&#8221; in <a href="git-rev-parse.html">git-rev-parse(1)</a> for details.
846</p>
847</li>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]848</ul></div>
849<div class="para"><p>The special case of restarting an incremental import from the
850current branch value should be written as:</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]851<div class="listingblock">
852<div class="content">
853<pre><tt> from refs/heads/branch^0</tt></pre>
854</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]855<div class="para"><p>The <tt>&#94;0</tt> suffix is necessary as fast-import does not permit a branch to
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]856start from itself, and the branch is created in memory before the
Junio C Hamano6926bef2007-06-16 09:54:05[diff] [blame]857<tt>from</tt> command is even read from the input. Adding <tt>&#94;0</tt> will force
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]858fast-import to resolve the commit through Git's revision parsing library,
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]859rather than its internal branch table, thereby loading in the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]860existing value of the branch.</p></div>
861<h4 id="_tt_merge_tt"><tt>merge</tt></h4>
862<div class="para"><p>Includes one additional ancestor commit. If the <tt>from</tt> command is
Junio C Hamano25c0b702008-03-23 09:41:37[diff] [blame]863omitted when creating a new branch, the first <tt>merge</tt> commit will be
864the first ancestor of the current commit, and the branch will start
865out with no files. An unlimited number of <tt>merge</tt> commands per
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]866commit are permitted by fast-import, thereby establishing an n-way merge.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]867However Git's other tools never create commits with more than 15
868additional ancestors (forming a 16-way merge). For this reason
869it is suggested that frontends do not use more than 15 <tt>merge</tt>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]870commands per commit; 16, if starting a new, empty branch.</p></div>
871<div class="para"><p>Here <tt>&lt;committish&gt;</tt> is any of the commit specification expressions
872also accepted by <tt>from</tt> (see above).</p></div>
873<h4 id="_tt_filemodify_tt"><tt>filemodify</tt></h4>
874<div class="para"><p>Included in a <tt>commit</tt> command to add a new file or change the
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]875content of an existing file. This command has two different means
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]876of specifying the content of the file.</p></div>
877<div class="vlist"><dl>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]878<dt>
879External data format
880</dt>
881<dd>
882<p>
883 The data content for the file was already supplied by a prior
884 <tt>blob</tt> command. The frontend just needs to connect it.
885</p>
886<div class="literalblock">
887<div class="content">
888<pre><tt> 'M' SP &lt;mode&gt; SP &lt;dataref&gt; SP &lt;path&gt; LF</tt></pre>
889</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]890<div class="para"><p>Here <tt>&lt;dataref&gt;</tt> can be either a mark reference (<tt>:&lt;idnum&gt;</tt>)
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]891set by a prior <tt>blob</tt> command, or a full 40-byte SHA-1 of an
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]892existing Git blob object.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]893</dd>
894<dt>
895Inline data format
896</dt>
897<dd>
898<p>
899 The data content for the file has not been supplied yet.
900 The frontend wants to supply it as part of this modify
901 command.
902</p>
903<div class="literalblock">
904<div class="content">
905<pre><tt> 'M' SP &lt;mode&gt; SP 'inline' SP &lt;path&gt; LF
906 data</tt></pre>
907</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]908<div class="para"><p>See below for a detailed description of the <tt>data</tt> command.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]909</dd>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]910</dl></div>
911<div class="para"><p>In both formats <tt>&lt;mode&gt;</tt> is the type of file entry, specified
912in octal. Git only supports the following modes:</p></div>
913<div class="ilist"><ul>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]914<li>
915<p>
916<tt>100644</tt> or <tt>644</tt>: A normal (not-executable) file. The majority
917 of files in most projects use this mode. If in doubt, this is
918 what you want.
919</p>
920</li>
921<li>
922<p>
923<tt>100755</tt> or <tt>755</tt>: A normal, but executable, file.
924</p>
925</li>
926<li>
927<p>
928<tt>120000</tt>: A symlink, the content of the file will be the link target.
929</p>
930</li>
Junio C Hamano915cd9b2008-07-20 01:24:17[diff] [blame]931<li>
932<p>
933<tt>160000</tt>: A gitlink, SHA-1 of the object refers to a commit in
934 another repository. Git links can only be specified by SHA or through
935 a commit mark. They are used to implement submodules.
936</p>
937</li>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]938</ul></div>
939<div class="para"><p>In both formats <tt>&lt;path&gt;</tt> is the complete path of the file to be added
940(if not already existing) or modified (if already existing).</p></div>
941<div class="para"><p>A <tt>&lt;path&gt;</tt> string must use UNIX-style directory separators (forward
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]942slash <tt>/</tt>), may contain any byte other than <tt>LF</tt>, and must not
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]943start with double quote (<tt>"</tt>).</p></div>
944<div class="para"><p>If an <tt>LF</tt> or double quote must be encoded into <tt>&lt;path&gt;</tt> shell-style
945quoting should be used, e.g. <tt>"path/with\n and \" in it"</tt>.</p></div>
946<div class="para"><p>The value of <tt>&lt;path&gt;</tt> must be in canonical form. That is it must not:</p></div>
947<div class="ilist"><ul>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]948<li>
949<p>
950contain an empty directory component (e.g. <tt>foo//bar</tt> is invalid),
951</p>
952</li>
953<li>
954<p>
Junio C Hamanoc51fede2007-03-12 07:29:20[diff] [blame]955end with a directory separator (e.g. <tt>foo/</tt> is invalid),
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]956</p>
957</li>
958<li>
959<p>
Junio C Hamanoc51fede2007-03-12 07:29:20[diff] [blame]960start with a directory separator (e.g. <tt>/foo</tt> is invalid),
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]961</p>
962</li>
963<li>
964<p>
965contain the special component <tt>.</tt> or <tt>..</tt> (e.g. <tt>foo/./bar</tt> and
966 <tt>foo/../bar</tt> are invalid).
967</p>
968</li>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]969</ul></div>
970<div class="para"><p>It is recommended that <tt>&lt;path&gt;</tt> always be encoded using UTF-8.</p></div>
971<h4 id="_tt_filedelete_tt"><tt>filedelete</tt></h4>
972<div class="para"><p>Included in a <tt>commit</tt> command to remove a file or recursively
Junio C Hamano06216df2007-07-10 07:49:37[diff] [blame]973delete an entire directory from the branch. If the file or directory
974removal makes its parent directory empty, the parent directory will
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]975be automatically removed too. This cascades up the tree until the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]976first non-empty directory or the root is reached.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]977<div class="literalblock">
978<div class="content">
979<pre><tt> 'D' SP &lt;path&gt; LF</tt></pre>
980</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]981<div class="para"><p>here <tt>&lt;path&gt;</tt> is the complete path of the file or subdirectory to
Junio C Hamano06216df2007-07-10 07:49:37[diff] [blame]982be removed from the branch.
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]983See <tt>filemodify</tt> above for a detailed description of <tt>&lt;path&gt;</tt>.</p></div>
984<h4 id="_tt_filecopy_tt"><tt>filecopy</tt></h4>
985<div class="para"><p>Recursively copies an existing file or subdirectory to a different
Junio C Hamanoc0ea7c62007-07-15 07:19:06[diff] [blame]986location within the branch. The existing file or directory must
987exist. If the destination exists it will be completely replaced
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]988by the content copied from the source.</p></div>
Junio C Hamanoc0ea7c62007-07-15 07:19:06[diff] [blame]989<div class="literalblock">
990<div class="content">
991<pre><tt> 'C' SP &lt;path&gt; SP &lt;path&gt; LF</tt></pre>
992</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]993<div class="para"><p>here the first <tt>&lt;path&gt;</tt> is the source location and the second
Junio C Hamanoc0ea7c62007-07-15 07:19:06[diff] [blame]994<tt>&lt;path&gt;</tt> is the destination. See <tt>filemodify</tt> above for a detailed
995description of what <tt>&lt;path&gt;</tt> may look like. To use a source path
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]996that contains SP the path must be quoted.</p></div>
997<div class="para"><p>A <tt>filecopy</tt> command takes effect immediately. Once the source
Junio C Hamanoc0ea7c62007-07-15 07:19:06[diff] [blame]998location has been copied to the destination any future commands
999applied to the source location will not impact the destination of
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1000the copy.</p></div>
1001<h4 id="_tt_filerename_tt"><tt>filerename</tt></h4>
1002<div class="para"><p>Renames an existing file or subdirectory to a different location
Junio C Hamano06216df2007-07-10 07:49:37[diff] [blame]1003within the branch. The existing file or directory must exist. If
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1004the destination exists it will be replaced by the source directory.</p></div>
Junio C Hamano06216df2007-07-10 07:49:37[diff] [blame]1005<div class="literalblock">
1006<div class="content">
1007<pre><tt> 'R' SP &lt;path&gt; SP &lt;path&gt; LF</tt></pre>
1008</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1009<div class="para"><p>here the first <tt>&lt;path&gt;</tt> is the source location and the second
Junio C Hamano06216df2007-07-10 07:49:37[diff] [blame]1010<tt>&lt;path&gt;</tt> is the destination. See <tt>filemodify</tt> above for a detailed
1011description of what <tt>&lt;path&gt;</tt> may look like. To use a source path
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1012that contains SP the path must be quoted.</p></div>
1013<div class="para"><p>A <tt>filerename</tt> command takes effect immediately. Once the source
Junio C Hamano06216df2007-07-10 07:49:37[diff] [blame]1014location has been renamed to the destination any future commands
1015applied to the source location will create new files there and not
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1016impact the destination of the rename.</p></div>
1017<div class="para"><p>Note that a <tt>filerename</tt> is the same as a <tt>filecopy</tt> followed by a
Junio C Hamanoc0ea7c62007-07-15 07:19:06[diff] [blame]1018<tt>filedelete</tt> of the source location. There is a slight performance
1019advantage to using <tt>filerename</tt>, but the advantage is so small
1020that it is never worth trying to convert a delete/add pair in
1021source material into a rename for fast-import. This <tt>filerename</tt>
1022command is provided just to simplify frontends that already have
1023rename information and don't want bother with decomposing it into a
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1024<tt>filecopy</tt> followed by a <tt>filedelete</tt>.</p></div>
1025<h4 id="_tt_filedeleteall_tt"><tt>filedeleteall</tt></h4>
1026<div class="para"><p>Included in a <tt>commit</tt> command to remove all files (and also all
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1027directories) from the branch. This command resets the internal
1028branch structure to have no files in it, allowing the frontend
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1029to subsequently add all interesting files from scratch.</p></div>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1030<div class="literalblock">
1031<div class="content">
1032<pre><tt> 'deleteall' LF</tt></pre>
1033</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1034<div class="para"><p>This command is extremely useful if the frontend does not know
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1035(or does not care to know) what files are currently on the branch,
1036and therefore cannot generate the proper <tt>filedelete</tt> commands to
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1037update the content.</p></div>
1038<div class="para"><p>Issuing a <tt>filedeleteall</tt> followed by the needed <tt>filemodify</tt>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1039commands to set the correct content will produce the same results
1040as sending only the needed <tt>filemodify</tt> and <tt>filedelete</tt> commands.
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1041The <tt>filedeleteall</tt> approach may however require fast-import to use slightly
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1042more memory per active branch (less than 1 MiB for even most large
1043projects); so frontends that can easily obtain only the affected
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1044paths for a commit are encouraged to do so.</p></div>
Junio C Hamano3b70d3c2009-11-21 17:37:37[diff] [blame]1045<h4 id="_tt_notemodify_tt"><tt>notemodify</tt></h4>
1046<div class="para"><p>Included in a <tt>commit</tt> command to add a new note (annotating a given
1047commit) or change the content of an existing note. This command has
1048two different means of specifying the content of the note.</p></div>
1049<div class="vlist"><dl>
1050<dt>
1051External data format
1052</dt>
1053<dd>
1054<p>
1055 The data content for the note was already supplied by a prior
1056 <tt>blob</tt> command. The frontend just needs to connect it to the
1057 commit that is to be annotated.
1058</p>
1059<div class="literalblock">
1060<div class="content">
1061<pre><tt> 'N' SP &lt;dataref&gt; SP &lt;committish&gt; LF</tt></pre>
1062</div></div>
1063<div class="para"><p>Here <tt>&lt;dataref&gt;</tt> can be either a mark reference (<tt>:&lt;idnum&gt;</tt>)
1064set by a prior <tt>blob</tt> command, or a full 40-byte SHA-1 of an
1065existing Git blob object.</p></div>
1066</dd>
1067<dt>
1068Inline data format
1069</dt>
1070<dd>
1071<p>
1072 The data content for the note has not been supplied yet.
1073 The frontend wants to supply it as part of this modify
1074 command.
1075</p>
1076<div class="literalblock">
1077<div class="content">
1078<pre><tt> 'N' SP 'inline' SP &lt;committish&gt; LF
1079 data</tt></pre>
1080</div></div>
1081<div class="para"><p>See below for a detailed description of the <tt>data</tt> command.</p></div>
1082</dd>
1083</dl></div>
1084<div class="para"><p>In both formats <tt>&lt;committish&gt;</tt> is any of the commit specification
1085expressions also accepted by <tt>from</tt> (see above).</p></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1086<h3 id="_tt_mark_tt"><tt>mark</tt></h3><div style="clear:left"></div>
1087<div class="para"><p>Arranges for fast-import to save a reference to the current object, allowing
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1088the frontend to recall this object at a future point in time, without
1089knowing its SHA-1. Here the current object is the object creation
1090command the <tt>mark</tt> command appears within. This can be <tt>commit</tt>,
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1091<tt>tag</tt>, and <tt>blob</tt>, but <tt>commit</tt> is the most common usage.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1092<div class="literalblock">
1093<div class="content">
1094<pre><tt> 'mark' SP ':' &lt;idnum&gt; LF</tt></pre>
1095</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1096<div class="para"><p>where <tt>&lt;idnum&gt;</tt> is the number assigned by the frontend to this mark.
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1097The value of <tt>&lt;idnum&gt;</tt> is expressed as an ASCII decimal integer.
1098The value 0 is reserved and cannot be used as
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1099a mark. Only values greater than or equal to 1 may be used as marks.</p></div>
1100<div class="para"><p>New marks are created automatically. Existing marks can be moved
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1101to another object simply by reusing the same <tt>&lt;idnum&gt;</tt> in another
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1102<tt>mark</tt> command.</p></div>
1103<h3 id="_tt_tag_tt"><tt>tag</tt></h3><div style="clear:left"></div>
1104<div class="para"><p>Creates an annotated tag referring to a specific commit. To create
1105lightweight (non-annotated) tags see the <tt>reset</tt> command below.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1106<div class="literalblock">
1107<div class="content">
1108<pre><tt> 'tag' SP &lt;name&gt; LF
1109 'from' SP &lt;committish&gt; LF
Junio C Hamano6d325df2010-01-01 00:02:47[diff] [blame]1110 'tagger' (SP &lt;name&gt;)? SP LT &lt;email&gt; GT SP &lt;when&gt; LF
Junio C Hamanoea5dae62007-05-10 23:25:08[diff] [blame]1111 data</tt></pre>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1112</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1113<div class="para"><p>where <tt>&lt;name&gt;</tt> is the name of the tag to create.</p></div>
1114<div class="para"><p>Tag names are automatically prefixed with <tt>refs/tags/</tt> when stored
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1115in Git, so importing the CVS branch symbol <tt>RELENG-1_0-FINAL</tt> would
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1116use just <tt>RELENG-1_0-FINAL</tt> for <tt>&lt;name&gt;</tt>, and fast-import will write the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1117corresponding ref as <tt>refs/tags/RELENG-1_0-FINAL</tt>.</p></div>
1118<div class="para"><p>The value of <tt>&lt;name&gt;</tt> must be a valid refname in Git and therefore
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1119may contain forward slashes. As <tt>LF</tt> is not valid in a Git refname,
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1120no quoting or escaping syntax is supported here.</p></div>
1121<div class="para"><p>The <tt>from</tt> command is the same as in the <tt>commit</tt> command; see
1122above for details.</p></div>
1123<div class="para"><p>The <tt>tagger</tt> command uses the same format as <tt>committer</tt> within
1124<tt>commit</tt>; again see above for details.</p></div>
1125<div class="para"><p>The <tt>data</tt> command following <tt>tagger</tt> must supply the annotated tag
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1126message (see below for <tt>data</tt> command syntax). To import an empty
1127tag message use a 0 length data. Tag messages are free-form and are
1128not interpreted by Git. Currently they must be encoded in UTF-8,
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1129as fast-import does not permit other encodings to be specified.</p></div>
1130<div class="para"><p>Signing annotated tags during import from within fast-import is not
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1131supported. Trying to include your own PGP/GPG signature is not
1132recommended, as the frontend does not (easily) have access to the
1133complete set of bytes which normally goes into such a signature.
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1134If signing is required, create lightweight tags from within fast-import with
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1135<tt>reset</tt>, then create the annotated versions of those tags offline
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]1136with the standard <em>git tag</em> process.</p></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1137<h3 id="_tt_reset_tt"><tt>reset</tt></h3><div style="clear:left"></div>
1138<div class="para"><p>Creates (or recreates) the named branch, optionally starting from
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1139a specific revision. The reset command allows a frontend to issue
1140a new <tt>from</tt> command for an existing branch, or to create a new
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1141branch from an existing commit without creating a new commit.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1142<div class="literalblock">
1143<div class="content">
1144<pre><tt> 'reset' SP &lt;ref&gt; LF
1145 ('from' SP &lt;committish&gt; LF)?
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1146 LF?</tt></pre>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1147</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1148<div class="para"><p>For a detailed description of <tt>&lt;ref&gt;</tt> and <tt>&lt;committish&gt;</tt> see above
1149under <tt>commit</tt> and <tt>from</tt>.</p></div>
1150<div class="para"><p>The <tt>LF</tt> after the command is optional (it used to be required).</p></div>
1151<div class="para"><p>The <tt>reset</tt> command can also be used to create lightweight
1152(non-annotated) tags. For example:</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1153<div class="exampleblock">
1154<div class="exampleblock-content">
1155<div class="literalblock">
1156<div class="content">
1157<pre><tt>reset refs/tags/938
1158from :938</tt></pre>
1159</div></div>
1160</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1161<div class="para"><p>would create the lightweight tag <tt>refs/tags/938</tt> referring to
1162whatever commit mark <tt>:938</tt> references.</p></div>
1163<h3 id="_tt_blob_tt"><tt>blob</tt></h3><div style="clear:left"></div>
1164<div class="para"><p>Requests writing one file revision to the packfile. The revision
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1165is not connected to any commit; this connection must be formed in
1166a subsequent <tt>commit</tt> command by referencing the blob through an
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1167assigned mark.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1168<div class="literalblock">
1169<div class="content">
1170<pre><tt> 'blob' LF
1171 mark?
1172 data</tt></pre>
1173</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1174<div class="para"><p>The mark command is optional here as some frontends have chosen
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1175to generate the Git SHA-1 for the blob on their own, and feed that
Junio C Hamano167b1382010-01-31 23:04:31[diff] [blame]1176directly to <tt>commit</tt>. This is typically more work than it's worth
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1177however, as marks are inexpensive to store and easy to use.</p></div>
1178<h3 id="_tt_data_tt"><tt>data</tt></h3><div style="clear:left"></div>
1179<div class="para"><p>Supplies raw data (for use as blob/file content, commit messages, or
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1180annotated tag messages) to fast-import. Data can be supplied using an exact
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1181byte count or delimited with a terminating line. Real frontends
1182intended for production-quality conversions should always use the
1183exact byte count format, as it is more robust and performs better.
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1184The delimited format is intended primarily for testing fast-import.</p></div>
1185<div class="para"><p>Comment lines appearing within the <tt>&lt;raw&gt;</tt> part of <tt>data</tt> commands
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1186are always taken to be part of the body of the data and are therefore
1187never ignored by fast-import. This makes it safe to import any
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1188file/message content whose lines might start with <tt>#</tt>.</p></div>
1189<div class="vlist"><dl>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1190<dt>
1191Exact byte count format
1192</dt>
1193<dd>
1194<p>
1195 The frontend must specify the number of bytes of data.
1196</p>
1197<div class="literalblock">
1198<div class="content">
1199<pre><tt> 'data' SP &lt;count&gt; LF
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1200 &lt;raw&gt; LF?</tt></pre>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1201</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1202<div class="para"><p>where <tt>&lt;count&gt;</tt> is the exact number of bytes appearing within
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1203<tt>&lt;raw&gt;</tt>. The value of <tt>&lt;count&gt;</tt> is expressed as an ASCII decimal
1204integer. The <tt>LF</tt> on either side of <tt>&lt;raw&gt;</tt> is not
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1205included in <tt>&lt;count&gt;</tt> and will not be included in the imported data.</p></div>
1206<div class="para"><p>The <tt>LF</tt> after <tt>&lt;raw&gt;</tt> is optional (it used to be required) but
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1207recommended. Always including it makes debugging a fast-import
1208stream easier as the next command always starts in column 0
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1209of the next line, even if <tt>&lt;raw&gt;</tt> did not end with an <tt>LF</tt>.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1210</dd>
1211<dt>
1212Delimited format
1213</dt>
1214<dd>
1215<p>
1216 A delimiter string is used to mark the end of the data.
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1217 fast-import will compute the length by searching for the delimiter.
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]1218 This format is primarily useful for testing and is not
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1219 recommended for real data.
1220</p>
1221<div class="literalblock">
1222<div class="content">
1223<pre><tt> 'data' SP '&lt;&lt;' &lt;delim&gt; LF
1224 &lt;raw&gt; LF
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1225 &lt;delim&gt; LF
1226 LF?</tt></pre>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1227</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1228<div class="para"><p>where <tt>&lt;delim&gt;</tt> is the chosen delimiter string. The string <tt>&lt;delim&gt;</tt>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1229must not appear on a line by itself within <tt>&lt;raw&gt;</tt>, as otherwise
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1230fast-import will think the data ends earlier than it really does. The <tt>LF</tt>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1231immediately trailing <tt>&lt;raw&gt;</tt> is part of <tt>&lt;raw&gt;</tt>. This is one of
1232the limitations of the delimited format, it is impossible to supply
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1233a data chunk which does not have an LF as its last byte.</p></div>
1234<div class="para"><p>The <tt>LF</tt> after <tt>&lt;delim&gt; LF</tt> is optional (it used to be required).</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1235</dd>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1236</dl></div>
1237<h3 id="_tt_checkpoint_tt"><tt>checkpoint</tt></h3><div style="clear:left"></div>
1238<div class="para"><p>Forces fast-import to close the current packfile, start a new one, and to
1239save out all current branch refs, tags and marks.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1240<div class="literalblock">
1241<div class="content">
1242<pre><tt> 'checkpoint' LF
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1243 LF?</tt></pre>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1244</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1245<div class="para"><p>Note that fast-import automatically switches packfiles when the current
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1246packfile reaches --max-pack-size, or 4 GiB, whichever limit is
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1247smaller. During an automatic packfile switch fast-import does not update
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1248the branch refs, tags or marks.</p></div>
1249<div class="para"><p>As a <tt>checkpoint</tt> can require a significant amount of CPU time and
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1250disk IO (to compute the overall pack SHA-1 checksum, generate the
1251corresponding index file, and update the refs) it can easily take
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1252several minutes for a single <tt>checkpoint</tt> command to complete.</p></div>
1253<div class="para"><p>Frontends may choose to issue checkpoints during extremely large
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1254and long running imports, or when they need to allow another Git
1255process access to a branch. However given that a 30 GiB Subversion
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1256repository can be loaded into Git through fast-import in about 3 hours,
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1257explicit checkpointing may not be necessary.</p></div>
1258<div class="para"><p>The <tt>LF</tt> after the command is optional (it used to be required).</p></div>
1259<h3 id="_tt_progress_tt"><tt>progress</tt></h3><div style="clear:left"></div>
1260<div class="para"><p>Causes fast-import to print the entire <tt>progress</tt> line unmodified to
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1261its standard output channel (file descriptor 1) when the command is
1262processed from the input stream. The command otherwise has no impact
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1263on the current import, or on any of fast-import's internal state.</p></div>
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1264<div class="literalblock">
1265<div class="content">
1266<pre><tt> 'progress' SP &lt;any&gt; LF
1267 LF?</tt></pre>
1268</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1269<div class="para"><p>The <tt>&lt;any&gt;</tt> part of the command may contain any sequence of bytes
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1270that does not contain <tt>LF</tt>. The <tt>LF</tt> after the command is optional.
1271Callers may wish to process the output through a tool such as sed to
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1272remove the leading part of the line, for example:</p></div>
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1273<div class="exampleblock">
1274<div class="exampleblock-content">
1275<div class="literalblock">
1276<div class="content">
Junio C Hamanofce7c7e2008-07-02 03:06:38[diff] [blame]1277<pre><tt>frontend | git fast-import | sed 's/^progress //'</tt></pre>
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1278</div></div>
1279</div></div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1280<div class="para"><p>Placing a <tt>progress</tt> command immediately after a <tt>checkpoint</tt> will
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1281inform the reader when the <tt>checkpoint</tt> has been completed and it
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1282can safely access the refs that fast-import updated.</p></div>
Junio C Hamano6ce6b6c2010-01-18 01:25:50[diff] [blame]1283<h3 id="_tt_feature_tt"><tt>feature</tt></h3><div style="clear:left"></div>
1284<div class="para"><p>Require that fast-import supports the specified feature, or abort if
1285it does not.</p></div>
1286<div class="literalblock">
1287<div class="content">
1288<pre><tt> 'feature' SP &lt;feature&gt; LF</tt></pre>
1289</div></div>
1290<div class="para"><p>The &lt;feature&gt; part of the command may be any string matching
1291^[a-zA-Z][a-zA-Z-]*$ and should be understood by fast-import.</p></div>
1292<div class="para"><p>Feature work identical as their option counterparts with the
1293exception of the import-marks feature, see below.</p></div>
1294<div class="para"><p>The following features are currently supported:</p></div>
1295<div class="ilist"><ul>
1296<li>
1297<p>
1298date-format
1299</p>
1300</li>
1301<li>
1302<p>
1303import-marks
1304</p>
1305</li>
1306<li>
1307<p>
1308export-marks
1309</p>
1310</li>
1311<li>
1312<p>
1313relative-marks
1314</p>
1315</li>
1316<li>
1317<p>
1318no-relative-marks
1319</p>
1320</li>
1321<li>
1322<p>
1323force
1324</p>
1325</li>
1326</ul></div>
1327<div class="para"><p>The import-marks behaves differently from when it is specified as
1328commandline option in that only one "feature import-marks" is allowed
1329per stream. Also, any --import-marks= specified on the commandline
1330will override those from the stream (if any).</p></div>
1331<h3 id="_tt_option_tt"><tt>option</tt></h3><div style="clear:left"></div>
1332<div class="para"><p>Processes the specified option so that git fast-import behaves in a
1333way that suits the frontend's needs.
1334Note that options specified by the frontend are overridden by any
1335options the user may specify to git fast-import itself.</p></div>
1336<div class="literalblock">
1337<div class="content">
1338<pre><tt> 'option' SP &lt;option&gt; LF</tt></pre>
1339</div></div>
1340<div class="para"><p>The <tt>&lt;option&gt;</tt> part of the command may contain any of the options
1341listed in the OPTIONS section that do not change import semantics,
1342without the leading <em>--</em> and is treated in the same way.</p></div>
1343<div class="para"><p>Option commands must be the first commands on the input (not counting
1344feature commands), to give an option command after any non-option
1345command is an error.</p></div>
1346<div class="para"><p>The following commandline options change import semantics and may therefore
1347not be passed as option:</p></div>
1348<div class="ilist"><ul>
1349<li>
1350<p>
1351date-format
1352</p>
1353</li>
1354<li>
1355<p>
1356import-marks
1357</p>
1358</li>
1359<li>
1360<p>
1361export-marks
1362</p>
1363</li>
1364<li>
1365<p>
1366force
1367</p>
1368</li>
1369</ul></div>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1370</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1371<h2 id="_crash_reports">Crash Reports</h2>
Junio C Hamano21391932008-02-16 10:02:54[diff] [blame]1372<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1373<div class="para"><p>If fast-import is supplied invalid input it will terminate with a
Junio C Hamano21391932008-02-16 10:02:54[diff] [blame]1374non-zero exit status and create a crash report in the top level of
1375the Git repository it was importing into. Crash reports contain
1376a snapshot of the internal fast-import state as well as the most
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1377recent commands that lead up to the crash.</p></div>
1378<div class="para"><p>All recent commands (including stream comments, file changes and
Junio C Hamano21391932008-02-16 10:02:54[diff] [blame]1379progress commands) are shown in the command history within the crash
1380report, but raw file data and commit messages are excluded from the
1381crash report. This exclusion saves space within the report file
1382and reduces the amount of buffering that fast-import must perform
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1383during execution.</p></div>
1384<div class="para"><p>After writing a crash report fast-import will close the current
Junio C Hamano21391932008-02-16 10:02:54[diff] [blame]1385packfile and export the marks table. This allows the frontend
1386developer to inspect the repository state and resume the import from
1387the point where it crashed. The modified branches and tags are not
1388updated during a crash, as the import did not complete successfully.
1389Branch and tag information can be found in the crash report and
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1390must be applied manually if the update is needed.</p></div>
1391<div class="para"><p>An example crash:</p></div>
Junio C Hamano21391932008-02-16 10:02:54[diff] [blame]1392<div class="exampleblock">
1393<div class="exampleblock-content">
1394<div class="literalblock">
1395<div class="content">
1396<pre><tt>$ cat &gt;in &lt;&lt;END_OF_INPUT
1397# my very first test commit
1398commit refs/heads/master
1399committer Shawn O. Pearce &lt;spearce&gt; 19283 -0400
1400# who is that guy anyway?
1401data &lt;&lt;EOF
1402this is my commit
1403EOF
1404M 644 inline .gitignore
1405data &lt;&lt;EOF
1406.gitignore
1407EOF
1408M 777 inline bob
1409END_OF_INPUT</tt></pre>
1410</div></div>
1411<div class="literalblock">
1412<div class="content">
Junio C Hamanofce7c7e2008-07-02 03:06:38[diff] [blame]1413<pre><tt>$ git fast-import &lt;in
Junio C Hamano21391932008-02-16 10:02:54[diff] [blame]1414fatal: Corrupt mode: M 777 inline bob
1415fast-import: dumping crash report to .git/fast_import_crash_8434</tt></pre>
1416</div></div>
1417<div class="literalblock">
1418<div class="content">
1419<pre><tt>$ cat .git/fast_import_crash_8434
1420fast-import crash report:
1421 fast-import process: 8434
1422 parent process : 1391
1423 at Sat Sep 1 00:58:12 2007</tt></pre>
1424</div></div>
1425<div class="literalblock">
1426<div class="content">
1427<pre><tt>fatal: Corrupt mode: M 777 inline bob</tt></pre>
1428</div></div>
1429<div class="literalblock">
1430<div class="content">
1431<pre><tt>Most Recent Commands Before Crash
1432---------------------------------
1433 # my very first test commit
1434 commit refs/heads/master
1435 committer Shawn O. Pearce &lt;spearce&gt; 19283 -0400
1436 # who is that guy anyway?
1437 data &lt;&lt;EOF
1438 M 644 inline .gitignore
1439 data &lt;&lt;EOF
1440* M 777 inline bob</tt></pre>
1441</div></div>
1442<div class="literalblock">
1443<div class="content">
1444<pre><tt>Active Branch LRU
1445-----------------
1446 active_branches = 1 cur, 5 max</tt></pre>
1447</div></div>
1448<div class="literalblock">
1449<div class="content">
1450<pre><tt>pos clock name
1451~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1452 1) 0 refs/heads/master</tt></pre>
1453</div></div>
1454<div class="literalblock">
1455<div class="content">
1456<pre><tt>Inactive Branches
1457-----------------
1458refs/heads/master:
1459 status : active loaded dirty
1460 tip commit : 0000000000000000000000000000000000000000
1461 old tree : 0000000000000000000000000000000000000000
1462 cur tree : 0000000000000000000000000000000000000000
1463 commit clock: 0
1464 last pack :</tt></pre>
1465</div></div>
1466<div class="literalblock">
1467<div class="content">
1468<pre><tt>-------------------
1469END OF CRASH REPORT</tt></pre>
1470</div></div>
1471</div></div>
1472</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1473<h2 id="_tips_and_tricks">Tips and Tricks</h2>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1474<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1475<div class="para"><p>The following tips and tricks have been collected from various
1476users of fast-import, and are offered here as suggestions.</p></div>
1477<h3 id="_use_one_mark_per_commit">Use One Mark Per Commit</h3><div style="clear:left"></div>
1478<div class="para"><p>When doing a repository conversion, use a unique mark per commit
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1479(<tt>mark :&lt;n&gt;</tt>) and supply the --export-marks option on the command
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1480line. fast-import will dump a file which lists every mark and the Git
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1481object SHA-1 that corresponds to it. If the frontend can tie
1482the marks back to the source repository, it is easy to verify the
1483accuracy and completeness of the import by comparing each Git
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1484commit to the corresponding source revision.</p></div>
1485<div class="para"><p>Coming from a system such as Perforce or Subversion this should be
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1486quite simple, as the fast-import mark can also be the Perforce changeset
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1487number or the Subversion revision number.</p></div>
1488<h3 id="_freely_skip_around_branches">Freely Skip Around Branches</h3><div style="clear:left"></div>
1489<div class="para"><p>Don't bother trying to optimize the frontend to stick to one branch
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1490at a time during an import. Although doing so might be slightly
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1491faster for fast-import, it tends to increase the complexity of the frontend
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1492code considerably.</p></div>
1493<div class="para"><p>The branch LRU builtin to fast-import tends to behave very well, and the
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1494cost of activating an inactive branch is so low that bouncing around
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1495between branches has virtually no impact on import performance.</p></div>
1496<h3 id="_handling_renames">Handling Renames</h3><div style="clear:left"></div>
1497<div class="para"><p>When importing a renamed file or directory, simply delete the old
Junio C Hamano9dd8bb02007-02-12 07:15:35[diff] [blame]1498name(s) and modify the new name(s) during the corresponding commit.
1499Git performs rename detection after-the-fact, rather than explicitly
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1500during a commit.</p></div>
1501<h3 id="_use_tag_fixup_branches">Use Tag Fixup Branches</h3><div style="clear:left"></div>
1502<div class="para"><p>Some other SCM systems let the user create a tag from multiple
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1503files which are not from the same commit/changeset. Or to create
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1504tags which are a subset of the files available in the repository.</p></div>
1505<div class="para"><p>Importing these tags as-is in Git is impossible without making at
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1506least one commit which &#8220;fixes up&#8221; the files to match the content
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1507of the tag. Use fast-import's <tt>reset</tt> command to reset a dummy branch
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1508outside of your normal branch space to the base commit for the tag,
1509then commit one or more file fixup commits, and finally tag the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1510dummy branch.</p></div>
1511<div class="para"><p>For example since all normal branches are stored under <tt>refs/heads/</tt>
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1512name the tag fixup branch <tt>TAG_FIXUP</tt>. This way it is impossible for
1513the fixup branch used by the importer to have namespace conflicts
1514with real branches imported from the source (the name <tt>TAG_FIXUP</tt>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1515is not <tt>refs/heads/TAG_FIXUP</tt>).</p></div>
1516<div class="para"><p>When committing fixups, consider using <tt>merge</tt> to connect the
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1517commit(s) which are supplying file revisions to the fixup branch.
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]1518Doing so will allow tools such as <em>git blame</em> to track
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1519through the real commit history and properly annotate the source
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1520files.</p></div>
1521<div class="para"><p>After fast-import terminates the frontend will need to do <tt>rm .git/TAG_FIXUP</tt>
1522to remove the dummy branch.</p></div>
1523<h3 id="_import_now_repack_later">Import Now, Repack Later</h3><div style="clear:left"></div>
1524<div class="para"><p>As soon as fast-import completes the Git repository is completely valid
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]1525and ready for use. Typically this takes only a very short time,
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1526even for considerably large projects (100,000+ commits).</p></div>
1527<div class="para"><p>However repacking the repository is necessary to improve data
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1528locality and access performance. It can also take hours on extremely
1529large projects (especially if -f and a large --window parameter is
1530used). Since repacking is safe to run alongside readers and writers,
1531run the repack in the background and let it finish when it finishes.
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1532There is no reason to wait to explore your new Git project!</p></div>
1533<div class="para"><p>If you choose to wait for the repack, don't try to run benchmarks
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1534or performance tests until repacking is completed. fast-import outputs
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1535suboptimal packfiles that are simply never seen in real use
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1536situations.</p></div>
1537<h3 id="_repacking_historical_data">Repacking Historical Data</h3><div style="clear:left"></div>
1538<div class="para"><p>If you are repacking very old imported data (e.g. older than the
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1539last year), consider expending some extra CPU time and supplying
Junio C Hamano1aa40d22010-01-21 17:46:43[diff] [blame]1540--window=50 (or higher) when you run <em>git repack</em>.
Junio C Hamanodfc4ce72007-02-07 23:17:29[diff] [blame]1541This will take longer, but will also produce a smaller packfile.
1542You only need to expend the effort once, and everyone using your
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1543project will benefit from the smaller repository.</p></div>
1544<h3 id="_include_some_progress_messages">Include Some Progress Messages</h3><div style="clear:left"></div>
1545<div class="para"><p>Every once in a while have your frontend emit a <tt>progress</tt> message
Junio C Hamanoe52cf782007-08-19 19:15:52[diff] [blame]1546to fast-import. The contents of the messages are entirely free-form,
1547so one suggestion would be to output the current month and year
1548each time the current commit date moves into the next month.
1549Your users will feel better knowing how much of the data stream
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1550has been processed.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1551</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1552<h2 id="_packfile_optimization">Packfile Optimization</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1553<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1554<div class="para"><p>When packing a blob fast-import always attempts to deltify against the last
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1555blob written. Unless specifically arranged for by the frontend,
1556this will probably not be a prior version of the same file, so the
1557generated delta will not be the smallest possible. The resulting
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1558packfile will be compressed, but will not be optimal.</p></div>
1559<div class="para"><p>Frontends which have efficient access to all revisions of a
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1560single file (for example reading an RCS/CVS ,v file) can choose
1561to supply all revisions of that file as a sequence of consecutive
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1562<tt>blob</tt> commands. This allows fast-import to deltify the different file
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1563revisions against each other, saving space in the final packfile.
1564Marks can be used to later identify individual file revisions during
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1565a sequence of <tt>commit</tt> commands.</p></div>
1566<div class="para"><p>The packfile(s) created by fast-import do not encourage good disk access
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1567patterns. This is caused by fast-import writing the data in the order
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1568it is received on standard input, while Git typically organizes
1569data within packfiles to make the most recent (current tip) data
1570appear before historical data. Git also clusters commits together,
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1571speeding up revision traversal through better cache locality.</p></div>
1572<div class="para"><p>For this reason it is strongly recommended that users repack the
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1573repository with <tt>git repack -a -d</tt> after fast-import completes, allowing
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1574Git to reorganize the packfiles for faster data access. If blob
1575deltas are suboptimal (see above) then also adding the <tt>-f</tt> option
1576to force recomputation of all deltas can significantly reduce the
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1577final packfile size (30-50% smaller can be quite typical).</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1578</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1579<h2 id="_memory_utilization">Memory Utilization</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1580<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1581<div class="para"><p>There are a number of factors which affect how much memory fast-import
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1582requires to perform an import. Like critical sections of core
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]1583Git, fast-import uses its own memory allocators to amortize any overheads
1584associated with malloc. In practice fast-import tends to amortize any
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1585malloc overheads to 0, due to its use of large block allocations.</p></div>
1586<h3 id="_per_object">per object</h3><div style="clear:left"></div>
1587<div class="para"><p>fast-import maintains an in-memory structure for every object written in
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1588this execution. On a 32 bit system the structure is 32 bytes,
1589on a 64 bit system the structure is 40 bytes (due to the larger
1590pointer sizes). Objects in the table are not deallocated until
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1591fast-import terminates. Importing 2 million objects on a 32 bit system
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1592will require approximately 64 MiB of memory.</p></div>
1593<div class="para"><p>The object table is actually a hashtable keyed on the object name
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1594(the unique SHA-1). This storage configuration allows fast-import to reuse
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1595an existing or already written object and avoid writing duplicates
1596to the output packfile. Duplicate blobs are surprisingly common
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1597in an import, typically due to branch merges in the source.</p></div>
1598<h3 id="_per_mark">per mark</h3><div style="clear:left"></div>
1599<div class="para"><p>Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1600bytes, depending on pointer size) per mark. Although the array
1601is sparse, frontends are still strongly encouraged to use marks
1602between 1 and n, where n is the total number of marks required for
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1603this import.</p></div>
1604<h3 id="_per_branch">per branch</h3><div style="clear:left"></div>
1605<div class="para"><p>Branches are classified as active and inactive. The memory usage
1606of the two classes is significantly different.</p></div>
1607<div class="para"><p>Inactive branches are stored in a structure which uses 96 or 120
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1608bytes (32 bit or 64 bit systems, respectively), plus the length of
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1609the branch name (typically under 200 bytes), per branch. fast-import will
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1610easily handle as many as 10,000 inactive branches in under 2 MiB
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1611of memory.</p></div>
1612<div class="para"><p>Active branches have the same overhead as inactive branches, but
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1613also contain copies of every tree that has been recently modified on
1614that branch. If subtree <tt>include</tt> has not been modified since the
1615branch became active, its contents will not be loaded into memory,
1616but if subtree <tt>src</tt> has been modified by a commit since the branch
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1617became active, then its contents will be loaded in memory.</p></div>
1618<div class="para"><p>As active branches store metadata about the files contained on that
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1619branch, their in-memory storage size can grow to a considerable size
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1620(see below).</p></div>
1621<div class="para"><p>fast-import automatically moves active branches to inactive status based on
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1622a simple least-recently-used algorithm. The LRU chain is updated on
1623each <tt>commit</tt> command. The maximum number of active branches can be
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1624increased or decreased on the command line with --active-branches=.</p></div>
1625<h3 id="_per_active_tree">per active tree</h3><div style="clear:left"></div>
1626<div class="para"><p>Trees (aka directories) use just 12 bytes of memory on top of the
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1627memory required for their entries (see &#8220;per active file&#8221; below).
Junio C Hamanoa6387422007-08-25 03:54:27[diff] [blame]1628The cost of a tree is virtually 0, as its overhead amortizes out
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1629over the individual file entries.</p></div>
1630<h3 id="_per_active_file_entry">per active file entry</h3><div style="clear:left"></div>
1631<div class="para"><p>Files (and pointers to subtrees) within active trees require 52 or 64
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1632bytes (32/64 bit platforms) per entry. To conserve space, file and
1633tree names are pooled in a common string table, allowing the filename
1634&#8220;Makefile&#8221; to use just 16 bytes (after including the string header
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1635overhead) no matter how many times it occurs within the project.</p></div>
1636<div class="para"><p>The active branch LRU, when coupled with the filename string pool
Junio C Hamanod3339982007-02-09 08:38:48[diff] [blame]1637and lazy loading of subtrees, allows fast-import to efficiently import
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1638projects with 2,000+ branches and 45,114+ files in a very limited
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1639memory footprint (less than 2.7 MiB per active branch).</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1640</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1641<h2 id="_author">Author</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1642<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1643<div class="para"><p>Written by Shawn O. Pearce &lt;spearce@spearce.org&gt;.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1644</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1645<h2 id="_documentation">Documentation</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1646<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1647<div class="para"><p>Documentation by Shawn O. Pearce &lt;spearce@spearce.org&gt;.</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1648</div>
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1649<h2 id="_git">GIT</h2>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1650<div class="sectionbody">
Junio C Hamanoba4b9282008-07-06 05:20:31[diff] [blame]1651<div class="para"><p>Part of the <a href="git.html">git(1)</a> suite</p></div>
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1652</div>
1653<div id="footer">
1654<div id="footer-text">
Junio C Hamanod3304322010-02-05 03:28:27[diff] [blame^]1655Last updated 2010-02-05 03:28:09 UTC
Junio C Hamanodf60f442007-02-07 05:52:37[diff] [blame]1656</div>
1657</div>
1658</body>
1659</html>