Junio C Hamano | b96f40a | 2024-08-01 00:57:25 | [diff] [blame^] | 1 | <!DOCTYPE html> |
| 2 | <html xmlns="http://www.w3.org/1999/xhtml" lang="en"> |
| 3 | <head> |
| 4 | <meta charset="UTF-8"/> |
| 5 | <meta http-equiv="X-UA-Compatible" content="IE=edge"/> |
| 6 | <meta name="viewport" content="width=device-width, initial-scale=1.0"/> |
| 7 | <meta name="generator" content="Asciidoctor 2.0.20"/> |
| 8 | <title>reftable</title> |
| 9 | <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"/> |
| 10 | <style> |
| 11 | /*! Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */ |
| 12 | /* Uncomment the following line when using as a custom stylesheet */ |
| 13 | /* @import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"; */ |
| 14 | html{font-family:sans-serif;-webkit-text-size-adjust:100%} |
| 15 | a{background:none} |
| 16 | a:focus{outline:thin dotted} |
| 17 | a:active,a:hover{outline:0} |
| 18 | h1{font-size:2em;margin:.67em 0} |
| 19 | b,strong{font-weight:bold} |
| 20 | abbr{font-size:.9em} |
| 21 | abbr[title]{cursor:help;border-bottom:1px dotted #dddddf;text-decoration:none} |
| 22 | dfn{font-style:italic} |
| 23 | hr{height:0} |
| 24 | mark{background:#ff0;color:#000} |
| 25 | code,kbd,pre,samp{font-family:monospace;font-size:1em} |
| 26 | pre{white-space:pre-wrap} |
| 27 | q{quotes:"\201C" "\201D" "\2018" "\2019"} |
| 28 | small{font-size:80%} |
| 29 | sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline} |
| 30 | sup{top:-.5em} |
| 31 | sub{bottom:-.25em} |
| 32 | img{border:0} |
| 33 | svg:not(:root){overflow:hidden} |
| 34 | figure{margin:0} |
| 35 | audio,video{display:inline-block} |
| 36 | audio:not([controls]){display:none;height:0} |
| 37 | fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em} |
| 38 | legend{border:0;padding:0} |
| 39 | button,input,select,textarea{font-family:inherit;font-size:100%;margin:0} |
| 40 | button,input{line-height:normal} |
| 41 | button,select{text-transform:none} |
| 42 | button,html input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer} |
| 43 | button[disabled],html input[disabled]{cursor:default} |
| 44 | input[type=checkbox],input[type=radio]{padding:0} |
| 45 | button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0} |
| 46 | textarea{overflow:auto;vertical-align:top} |
| 47 | table{border-collapse:collapse;border-spacing:0} |
| 48 | *,::before,::after{box-sizing:border-box} |
| 49 | html,body{font-size:100%} |
| 50 | body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;line-height:1;position:relative;cursor:auto;-moz-tab-size:4;-o-tab-size:4;tab-size:4;word-wrap:anywhere;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased} |
| 51 | a:hover{cursor:pointer} |
| 52 | img,object,embed{max-width:100%;height:auto} |
| 53 | object,embed{height:100%} |
| 54 | img{-ms-interpolation-mode:bicubic} |
| 55 | .left{float:left!important} |
| 56 | .right{float:right!important} |
| 57 | .text-left{text-align:left!important} |
| 58 | .text-right{text-align:right!important} |
| 59 | .text-center{text-align:center!important} |
| 60 | .text-justify{text-align:justify!important} |
| 61 | .hide{display:none} |
| 62 | img,object,svg{display:inline-block;vertical-align:middle} |
| 63 | textarea{height:auto;min-height:50px} |
| 64 | select{width:100%} |
| 65 | .subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em} |
| 66 | div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0} |
| 67 | a{color:#2156a5;text-decoration:underline;line-height:inherit} |
| 68 | a:hover,a:focus{color:#1d4b8f} |
| 69 | a img{border:0} |
| 70 | p{line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility} |
| 71 | p aside{font-size:.875em;line-height:1.35;font-style:italic} |
| 72 | h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em} |
| 73 | h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0} |
| 74 | h1{font-size:2.125em} |
| 75 | h2{font-size:1.6875em} |
| 76 | h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em} |
| 77 | h4,h5{font-size:1.125em} |
| 78 | h6{font-size:1em} |
| 79 | hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em} |
| 80 | em,i{font-style:italic;line-height:inherit} |
| 81 | strong,b{font-weight:bold;line-height:inherit} |
| 82 | small{font-size:60%;line-height:inherit} |
| 83 | code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)} |
| 84 | ul,ol,dl{line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit} |
| 85 | ul,ol{margin-left:1.5em} |
| 86 | ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0} |
| 87 | ul.circle{list-style-type:circle} |
| 88 | ul.disc{list-style-type:disc} |
| 89 | ul.square{list-style-type:square} |
| 90 | ul.circle ul:not([class]),ul.disc ul:not([class]),ul.square ul:not([class]){list-style:inherit} |
| 91 | ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0} |
| 92 | dl dt{margin-bottom:.3125em;font-weight:bold} |
| 93 | dl dd{margin-bottom:1.25em} |
| 94 | blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd} |
| 95 | blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)} |
| 96 | @media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2} |
| 97 | h1{font-size:2.75em} |
| 98 | h2{font-size:2.3125em} |
| 99 | h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em} |
| 100 | h4{font-size:1.4375em}} |
| 101 | table{background:#fff;margin-bottom:1.25em;border:1px solid #dedede;word-wrap:normal} |
| 102 | table thead,table tfoot{background:#f7f8f7} |
| 103 | table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left} |
| 104 | table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)} |
| 105 | table tr.even,table tr.alt{background:#f8f8f7} |
| 106 | table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{line-height:1.6} |
| 107 | h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em} |
| 108 | h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400} |
| 109 | .center{margin-left:auto;margin-right:auto} |
| 110 | .stretch{width:100%} |
| 111 | .clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table} |
| 112 | .clearfix::after,.float-group::after{clear:both} |
| 113 | :not(pre).nobreak{word-wrap:normal} |
| 114 | :not(pre).nowrap{white-space:nowrap} |
| 115 | :not(pre).pre-wrap{white-space:pre-wrap} |
| 116 | :not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed} |
| 117 | pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed} |
| 118 | pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit} |
| 119 | pre>code{display:block} |
| 120 | pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal} |
| 121 | em em{font-style:normal} |
| 122 | strong strong{font-weight:400} |
| 123 | .keyseq{color:rgba(51,51,51,.8)} |
| 124 | kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;border-radius:3px;box-shadow:0 1px 0 rgba(0,0,0,.2),inset 0 0 0 .1em #fff;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap} |
| 125 | .keyseq kbd:first-child{margin-left:0} |
| 126 | .keyseq kbd:last-child{margin-right:0} |
| 127 | .menuseq,.menuref{color:#000} |
| 128 | .menuseq b:not(.caret),.menuref{font-weight:inherit} |
| 129 | .menuseq{word-spacing:-.02em} |
| 130 | .menuseq b.caret{font-size:1.25em;line-height:.8} |
| 131 | .menuseq i.caret{font-weight:bold;text-align:center;width:.45em} |
| 132 | b.button::before,b.button::after{position:relative;top:-1px;font-weight:400} |
| 133 | b.button::before{content:"[";padding:0 3px 0 2px} |
| 134 | b.button::after{content:"]";padding:0 2px 0 3px} |
| 135 | p a>code:hover{color:rgba(0,0,0,.9)} |
| 136 | #header,#content,#footnotes,#footer{width:100%;margin:0 auto;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em} |
| 137 | #header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table} |
| 138 | #header::after,#content::after,#footnotes::after,#footer::after{clear:both} |
| 139 | #content{margin-top:1.25em} |
| 140 | #content::before{content:none} |
| 141 | #header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0} |
| 142 | #header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf} |
| 143 | #header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px} |
| 144 | #header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:flex;flex-flow:row wrap} |
| 145 | #header .details span:first-child{margin-left:-.125em} |
| 146 | #header .details span.email a{color:rgba(0,0,0,.85)} |
| 147 | #header .details br{display:none} |
| 148 | #header .details br+span::before{content:"\00a0\2013\00a0"} |
| 149 | #header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)} |
| 150 | #header .details br+span#revremark::before{content:"\00a0|\00a0"} |
| 151 | #header #revnumber{text-transform:capitalize} |
| 152 | #header #revnumber::after{content:"\00a0"} |
| 153 | #content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem} |
| 154 | #toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em} |
| 155 | #toc>ul{margin-left:.125em} |
| 156 | #toc ul.sectlevel0>li>a{font-style:italic} |
| 157 | #toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0} |
| 158 | #toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none} |
| 159 | #toc li{line-height:1.3334;margin-top:.3334em} |
| 160 | #toc a{text-decoration:none} |
| 161 | #toc a:active{text-decoration:underline} |
| 162 | #toctitle{color:#7a2518;font-size:1.2em} |
| 163 | @media screen and (min-width:768px){#toctitle{font-size:1.375em} |
| 164 | body.toc2{padding-left:15em;padding-right:0} |
| 165 | #toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto} |
| 166 | #toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em} |
| 167 | #toc.toc2>ul{font-size:.9em;margin-bottom:0} |
| 168 | #toc.toc2 ul ul{margin-left:0;padding-left:1em} |
| 169 | #toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em} |
| 170 | body.toc2.toc-right{padding-left:0;padding-right:15em} |
| 171 | body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}} |
| 172 | @media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0} |
| 173 | #toc.toc2{width:20em} |
| 174 | #toc.toc2 #toctitle{font-size:1.375em} |
| 175 | #toc.toc2>ul{font-size:.95em} |
| 176 | #toc.toc2 ul ul{padding-left:1.25em} |
| 177 | body.toc2.toc-right{padding-left:0;padding-right:20em}} |
| 178 | #content #toc{border:1px solid #e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;border-radius:4px} |
| 179 | #content #toc>:first-child{margin-top:0} |
| 180 | #content #toc>:last-child{margin-bottom:0} |
| 181 | #footer{max-width:none;background:rgba(0,0,0,.8);padding:1.25em} |
| 182 | #footer-text{color:hsla(0,0%,100%,.8);line-height:1.44} |
| 183 | #content{margin-bottom:.625em} |
| 184 | .sect1{padding-bottom:.625em} |
| 185 | @media screen and (min-width:768px){#content{margin-bottom:1.25em} |
| 186 | .sect1{padding-bottom:1.25em}} |
| 187 | .sect1:last-child{padding-bottom:0} |
| 188 | .sect1+.sect1{border-top:1px solid #e7e7e9} |
| 189 | #content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400} |
| 190 | #content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em} |
| 191 | #content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible} |
| 192 | #content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none} |
| 193 | #content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221} |
| 194 | details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em} |
| 195 | details{margin-left:1.25rem} |
| 196 | details>summary{cursor:pointer;display:block;position:relative;line-height:1.6;margin-bottom:.625rem;outline:none;-webkit-tap-highlight-color:transparent} |
| 197 | details>summary::-webkit-details-marker{display:none} |
| 198 | details>summary::before{content:"";border:solid transparent;border-left:solid;border-width:.3em 0 .3em .5em;position:absolute;top:.5em;left:-1.25rem;transform:translateX(15%)} |
| 199 | details[open]>summary::before{border:solid transparent;border-top:solid;border-width:.5em .3em 0;transform:translateY(15%)} |
| 200 | details>summary::after{content:"";width:1.25rem;height:1em;position:absolute;top:.3em;left:-1.25rem} |
| 201 | .admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic} |
| 202 | table.tableblock.fit-content>caption.title{white-space:nowrap;width:0} |
| 203 | .paragraph.lead>p,#preamble>.sectionbody>[class=paragraph]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)} |
| 204 | .admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%} |
| 205 | .admonitionblock>table td.icon{text-align:center;width:80px} |
| 206 | .admonitionblock>table td.icon img{max-width:none} |
| 207 | .admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase} |
| 208 | .admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6);word-wrap:anywhere} |
| 209 | .admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0} |
| 210 | .exampleblock>.content{border:1px solid #e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;border-radius:4px} |
| 211 | .sidebarblock{border:1px solid #dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;border-radius:4px} |
| 212 | .sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center} |
| 213 | .exampleblock>.content>:first-child,.sidebarblock>.content>:first-child{margin-top:0} |
| 214 | .exampleblock>.content>:last-child,.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0} |
| 215 | .literalblock pre,.listingblock>.content>pre{border-radius:4px;overflow-x:auto;padding:1em;font-size:.8125em} |
| 216 | @media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}} |
| 217 | @media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}} |
| 218 | .literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class=highlight],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8} |
| 219 | .literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)} |
| 220 | .listingblock>.content{position:relative} |
| 221 | .listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5} |
| 222 | .listingblock:hover code[data-lang]::before{display:block} |
| 223 | .listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5} |
| 224 | .listingblock.terminal pre .command:not([data-prompt])::before{content:"$"} |
| 225 | .listingblock pre.highlightjs{padding:0} |
| 226 | .listingblock pre.highlightjs>code{padding:1em;border-radius:4px} |
| 227 | .listingblock pre.prettyprint{border-width:0} |
| 228 | .prettyprint{background:#f7f7f8} |
| 229 | pre.prettyprint .linenums{line-height:1.45;margin-left:2em} |
| 230 | pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0} |
| 231 | pre.prettyprint li code[data-lang]::before{opacity:1} |
| 232 | pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none} |
| 233 | table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none} |
| 234 | table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal} |
| 235 | table.linenotable td.code{padding-left:.75em} |
| 236 | table.linenotable td.linenos,pre.pygments .linenos{border-right:1px solid;opacity:.35;padding-right:.5em;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none} |
| 237 | pre.pygments span.linenos{display:inline-block;margin-right:.75em} |
| 238 | .quoteblock{margin:0 1em 1.25em 1.5em;display:table} |
| 239 | .quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em} |
| 240 | .quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify} |
| 241 | .quoteblock blockquote{margin:0;padding:0;border:0} |
| 242 | .quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)} |
| 243 | .quoteblock blockquote>.paragraph:last-child p{margin-bottom:0} |
| 244 | .quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right} |
| 245 | .verseblock{margin:0 1em 1.25em} |
| 246 | .verseblock pre{font-family:"Open Sans","DejaVu Sans",sans-serif;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility} |
| 247 | .verseblock pre strong{font-weight:400} |
| 248 | .verseblock .attribution{margin-top:1.25rem;margin-left:.5ex} |
| 249 | .quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic} |
| 250 | .quoteblock .attribution br,.verseblock .attribution br{display:none} |
| 251 | .quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)} |
| 252 | .quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none} |
| 253 | .quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0} |
| 254 | .quoteblock.abstract{margin:0 1em 1.25em;display:block} |
| 255 | .quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center} |
| 256 | .quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf} |
| 257 | .quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0} |
| 258 | .quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem} |
| 259 | .quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;font-size:.85rem;text-align:left;margin-right:0} |
| 260 | p.tableblock:last-child{margin-bottom:0} |
| 261 | td.tableblock>.content{margin-bottom:1.25em;word-wrap:anywhere} |
| 262 | td.tableblock>.content>:last-child{margin-bottom:-1.25em} |
| 263 | table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede} |
| 264 | table.grid-all>*>tr>*{border-width:1px} |
| 265 | table.grid-cols>*>tr>*{border-width:0 1px} |
| 266 | table.grid-rows>*>tr>*{border-width:1px 0} |
| 267 | table.frame-all{border-width:1px} |
| 268 | table.frame-ends{border-width:1px 0} |
| 269 | table.frame-sides{border-width:0 1px} |
| 270 | table.frame-none>colgroup+*>:first-child>*,table.frame-sides>colgroup+*>:first-child>*{border-top-width:0} |
| 271 | table.frame-none>:last-child>:last-child>*,table.frame-sides>:last-child>:last-child>*{border-bottom-width:0} |
| 272 | table.frame-none>*>tr>:first-child,table.frame-ends>*>tr>:first-child{border-left-width:0} |
| 273 | table.frame-none>*>tr>:last-child,table.frame-ends>*>tr>:last-child{border-right-width:0} |
| 274 | table.stripes-all>*>tr,table.stripes-odd>*>tr:nth-of-type(odd),table.stripes-even>*>tr:nth-of-type(even),table.stripes-hover>*>tr:hover{background:#f8f8f7} |
| 275 | th.halign-left,td.halign-left{text-align:left} |
| 276 | th.halign-right,td.halign-right{text-align:right} |
| 277 | th.halign-center,td.halign-center{text-align:center} |
| 278 | th.valign-top,td.valign-top{vertical-align:top} |
| 279 | th.valign-bottom,td.valign-bottom{vertical-align:bottom} |
| 280 | th.valign-middle,td.valign-middle{vertical-align:middle} |
| 281 | table thead th,table tfoot th{font-weight:bold} |
| 282 | tbody tr th{background:#f7f8f7} |
| 283 | tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold} |
| 284 | p.tableblock>code:only-child{background:none;padding:0} |
| 285 | p.tableblock{font-size:1em} |
| 286 | ol{margin-left:1.75em} |
| 287 | ul li ol{margin-left:1.5em} |
| 288 | dl dd{margin-left:1.125em} |
| 289 | dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0} |
| 290 | li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em} |
| 291 | ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none} |
| 292 | ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em} |
| 293 | ul.unstyled,ol.unstyled{margin-left:0} |
| 294 | li>p:empty:only-child::before{content:"";display:inline-block} |
| 295 | ul.checklist>li>p:first-child{margin-left:-1em} |
| 296 | ul.checklist>li>p:first-child>.fa-square-o:first-child,ul.checklist>li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em} |
| 297 | ul.checklist>li>p:first-child>input[type=checkbox]:first-child{margin-right:.25em} |
| 298 | ul.inline{display:flex;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em} |
| 299 | ul.inline>li{margin-left:1.25em} |
| 300 | .unstyled dl dt{font-weight:400;font-style:normal} |
| 301 | ol.arabic{list-style-type:decimal} |
| 302 | ol.decimal{list-style-type:decimal-leading-zero} |
| 303 | ol.loweralpha{list-style-type:lower-alpha} |
| 304 | ol.upperalpha{list-style-type:upper-alpha} |
| 305 | ol.lowerroman{list-style-type:lower-roman} |
| 306 | ol.upperroman{list-style-type:upper-roman} |
| 307 | ol.lowergreek{list-style-type:lower-greek} |
| 308 | .hdlist>table,.colist>table{border:0;background:none} |
| 309 | .hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none} |
| 310 | td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em} |
| 311 | td.hdlist1{font-weight:bold;padding-bottom:1.25em} |
| 312 | td.hdlist2{word-wrap:anywhere} |
| 313 | .literalblock+.colist,.listingblock+.colist{margin-top:-.5em} |
| 314 | .colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top} |
| 315 | .colist td:not([class]):first-child img{max-width:none} |
| 316 | .colist td:not([class]):last-child{padding:.25em 0} |
| 317 | .thumb,.th{line-height:0;display:inline-block;border:4px solid #fff;box-shadow:0 0 0 1px #ddd} |
| 318 | .imageblock.left{margin:.25em .625em 1.25em 0} |
| 319 | .imageblock.right{margin:.25em 0 1.25em .625em} |
| 320 | .imageblock>.title{margin-bottom:0} |
| 321 | .imageblock.thumb,.imageblock.th{border-width:6px} |
| 322 | .imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em} |
| 323 | .image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0} |
| 324 | .image.left{margin-right:.625em} |
| 325 | .image.right{margin-left:.625em} |
| 326 | a.image{text-decoration:none;display:inline-block} |
| 327 | a.image object{pointer-events:none} |
| 328 | sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super} |
| 329 | sup.footnote a,sup.footnoteref a{text-decoration:none} |
| 330 | sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline} |
| 331 | #footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em} |
| 332 | #footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0} |
| 333 | #footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em} |
| 334 | #footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em} |
| 335 | #footnotes .footnote:last-of-type{margin-bottom:0} |
| 336 | #content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0} |
| 337 | div.unbreakable{page-break-inside:avoid} |
| 338 | .big{font-size:larger} |
| 339 | .small{font-size:smaller} |
| 340 | .underline{text-decoration:underline} |
| 341 | .overline{text-decoration:overline} |
| 342 | .line-through{text-decoration:line-through} |
| 343 | .aqua{color:#00bfbf} |
| 344 | .aqua-background{background:#00fafa} |
| 345 | .black{color:#000} |
| 346 | .black-background{background:#000} |
| 347 | .blue{color:#0000bf} |
| 348 | .blue-background{background:#0000fa} |
| 349 | .fuchsia{color:#bf00bf} |
| 350 | .fuchsia-background{background:#fa00fa} |
| 351 | .gray{color:#606060} |
| 352 | .gray-background{background:#7d7d7d} |
| 353 | .green{color:#006000} |
| 354 | .green-background{background:#007d00} |
| 355 | .lime{color:#00bf00} |
| 356 | .lime-background{background:#00fa00} |
| 357 | .maroon{color:#600000} |
| 358 | .maroon-background{background:#7d0000} |
| 359 | .navy{color:#000060} |
| 360 | .navy-background{background:#00007d} |
| 361 | .olive{color:#606000} |
| 362 | .olive-background{background:#7d7d00} |
| 363 | .purple{color:#600060} |
| 364 | .purple-background{background:#7d007d} |
| 365 | .red{color:#bf0000} |
| 366 | .red-background{background:#fa0000} |
| 367 | .silver{color:#909090} |
| 368 | .silver-background{background:#bcbcbc} |
| 369 | .teal{color:#006060} |
| 370 | .teal-background{background:#007d7d} |
| 371 | .white{color:#bfbfbf} |
| 372 | .white-background{background:#fafafa} |
| 373 | .yellow{color:#bfbf00} |
| 374 | .yellow-background{background:#fafa00} |
| 375 | span.icon>.fa{cursor:default} |
| 376 | a span.icon>.fa{cursor:inherit} |
| 377 | .admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default} |
| 378 | .admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c} |
| 379 | .admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111} |
| 380 | .admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900} |
| 381 | .admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400} |
| 382 | .admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000} |
| 383 | .conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);border-radius:50%;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold} |
| 384 | .conum[data-value] *{color:#fff!important} |
| 385 | .conum[data-value]+b{display:none} |
| 386 | .conum[data-value]::after{content:attr(data-value)} |
| 387 | pre .conum[data-value]{position:relative;top:-.125em} |
| 388 | b.conum *{color:inherit!important} |
| 389 | .conum:not([data-value]):empty{display:none} |
| 390 | dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility} |
| 391 | h1,h2,p,td.content,span.alt,summary{letter-spacing:-.01em} |
| 392 | p strong,td.content strong,div.footnote strong{letter-spacing:-.005em} |
| 393 | p,blockquote,dt,td.content,td.hdlist1,span.alt,summary{font-size:1.0625rem} |
| 394 | p{margin-bottom:1.25rem} |
| 395 | .sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em} |
| 396 | .exampleblock>.content{background:#fffef7;border-color:#e0e0dc;box-shadow:0 1px 4px #e0e0dc} |
| 397 | .print-only{display:none!important} |
| 398 | @page{margin:1.25cm .75cm} |
| 399 | @media print{*{box-shadow:none!important;text-shadow:none!important} |
| 400 | html{font-size:80%} |
| 401 | a{color:inherit!important;text-decoration:underline!important} |
| 402 | a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important} |
| 403 | a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em} |
| 404 | abbr[title]{border-bottom:1px dotted} |
| 405 | abbr[title]::after{content:" (" attr(title) ")"} |
| 406 | pre,blockquote,tr,img,object,svg{page-break-inside:avoid} |
| 407 | thead{display:table-header-group} |
| 408 | svg{max-width:100%} |
| 409 | p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3} |
| 410 | h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid} |
| 411 | #header,#content,#footnotes,#footer{max-width:none} |
| 412 | #toc,.sidebarblock,.exampleblock>.content{background:none!important} |
| 413 | #toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important} |
| 414 | body.book #header{text-align:center} |
| 415 | body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em} |
| 416 | body.book #header .details{border:0!important;display:block;padding:0!important} |
| 417 | body.book #header .details span:first-child{margin-left:0!important} |
| 418 | body.book #header .details br{display:block} |
| 419 | body.book #header .details br+span::before{content:none!important} |
| 420 | body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important} |
| 421 | body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always} |
| 422 | .listingblock code[data-lang]::before{display:block} |
| 423 | #footer{padding:0 .9375em} |
| 424 | .hide-on-print{display:none!important} |
| 425 | .print-only{display:block!important} |
| 426 | .hide-for-print{display:none!important} |
| 427 | .show-for-print{display:inherit!important}} |
| 428 | @media amzn-kf8,print{#header>h1:first-child{margin-top:1.25rem} |
| 429 | .sect1{padding:0!important} |
| 430 | .sect1+.sect1{border:0} |
| 431 | #footer{background:none} |
| 432 | #footer-text{color:rgba(0,0,0,.6);font-size:.9em}} |
| 433 | @media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}} |
| 434 | </style> |
| 435 | </head> |
| 436 | <body class="article"> |
| 437 | <div id="header"> |
| 438 | </div> |
| 439 | <div id="content"> |
| 440 | <div class="sect1"> |
| 441 | <h2 id="_reftable">reftable</h2> |
| 442 | <div class="sectionbody"> |
| 443 | <div class="sect2"> |
| 444 | <h3 id="_overview">Overview</h3> |
| 445 | <div class="sect3"> |
| 446 | <h4 id="_problem_statement">Problem statement</h4> |
| 447 | <div class="paragraph"> |
| 448 | <p>Some repositories contain a lot of references (e.g. android at 866k, |
| 449 | rails at 31k). The existing packed-refs format takes up a lot of space |
| 450 | (e.g. 62M), and does not scale with additional references. Lookup of a |
| 451 | single reference requires linearly scanning the file.</p> |
| 452 | </div> |
| 453 | <div class="paragraph"> |
| 454 | <p>Atomic pushes modifying multiple references require copying the entire |
| 455 | packed-refs file, which can be a considerable amount of data moved |
| 456 | (e.g. 62M in, 62M out) for even small transactions (2 refs modified).</p> |
| 457 | </div> |
| 458 | <div class="paragraph"> |
| 459 | <p>Repositories with many loose references occupy a large number of disk |
| 460 | blocks from the local file system, as each reference is its own file |
| 461 | storing 41 bytes (and another file for the corresponding reflog). This |
| 462 | negatively affects the number of inodes available when a large number of |
| 463 | repositories are stored on the same filesystem. Readers can be penalized |
| 464 | due to the larger number of syscalls required to traverse and read the |
| 465 | <code>$GIT_DIR/refs</code> directory.</p> |
| 466 | </div> |
| 467 | </div> |
| 468 | <div class="sect3"> |
| 469 | <h4 id="_objectives">Objectives</h4> |
| 470 | <div class="ulist"> |
| 471 | <ul> |
| 472 | <li> |
| 473 | <p>Near constant time lookup for any single reference, even when the |
| 474 | repository is cold and not in process or kernel cache.</p> |
| 475 | </li> |
| 476 | <li> |
| 477 | <p>Near constant time verification if an object name is referred to by at least |
| 478 | one reference (for allow-tip-sha1-in-want).</p> |
| 479 | </li> |
| 480 | <li> |
| 481 | <p>Efficient enumeration of an entire namespace, such as <code>refs/tags/</code>.</p> |
| 482 | </li> |
| 483 | <li> |
| 484 | <p>Support atomic push with <code>O(size_of_update)</code> operations.</p> |
| 485 | </li> |
| 486 | <li> |
| 487 | <p>Combine reflog storage with ref storage for small transactions.</p> |
| 488 | </li> |
| 489 | <li> |
| 490 | <p>Separate reflog storage for base refs and historical logs.</p> |
| 491 | </li> |
| 492 | </ul> |
| 493 | </div> |
| 494 | </div> |
| 495 | <div class="sect3"> |
| 496 | <h4 id="_description">Description</h4> |
| 497 | <div class="paragraph"> |
| 498 | <p>A reftable file is a portable binary file format customized for |
| 499 | reference storage. References are sorted, enabling linear scans, binary |
| 500 | search lookup, and range scans.</p> |
| 501 | </div> |
| 502 | <div class="paragraph"> |
| 503 | <p>Storage in the file is organized into variable sized blocks. Prefix |
| 504 | compression is used within a single block to reduce disk space. Block |
| 505 | size and alignment are tunable by the writer.</p> |
| 506 | </div> |
| 507 | </div> |
| 508 | <div class="sect3"> |
| 509 | <h4 id="_performance">Performance</h4> |
| 510 | <div class="paragraph"> |
| 511 | <p>Space used, packed-refs vs. reftable:</p> |
| 512 | </div> |
| 513 | <table class="tableblock frame-all grid-all stretch"> |
| 514 | <colgroup> |
| 515 | <col style="width: 16.6666%;"/> |
| 516 | <col style="width: 16.6666%;"/> |
| 517 | <col style="width: 16.6666%;"/> |
| 518 | <col style="width: 16.6666%;"/> |
| 519 | <col style="width: 16.6666%;"/> |
| 520 | <col style="width: 16.667%;"/> |
| 521 | </colgroup> |
| 522 | <thead> |
| 523 | <tr> |
| 524 | <th class="tableblock halign-left valign-top">repository</th> |
| 525 | <th class="tableblock halign-right valign-top">packed-refs</th> |
| 526 | <th class="tableblock halign-right valign-top">reftable</th> |
| 527 | <th class="tableblock halign-right valign-top">% original</th> |
| 528 | <th class="tableblock halign-right valign-top">avg ref</th> |
| 529 | <th class="tableblock halign-right valign-top">avg obj</th> |
| 530 | </tr> |
| 531 | </thead> |
| 532 | <tbody> |
| 533 | <tr> |
| 534 | <td class="tableblock halign-left valign-top"><p class="tableblock">android</p></td> |
| 535 | <td class="tableblock halign-right valign-top"><p class="tableblock">62.2 M</p></td> |
| 536 | <td class="tableblock halign-right valign-top"><p class="tableblock">36.1 M</p></td> |
| 537 | <td class="tableblock halign-right valign-top"><p class="tableblock">58.0%</p></td> |
| 538 | <td class="tableblock halign-right valign-top"><p class="tableblock">33 bytes</p></td> |
| 539 | <td class="tableblock halign-right valign-top"><p class="tableblock">5 bytes</p></td> |
| 540 | </tr> |
| 541 | <tr> |
| 542 | <td class="tableblock halign-left valign-top"><p class="tableblock">rails</p></td> |
| 543 | <td class="tableblock halign-right valign-top"><p class="tableblock">1.8 M</p></td> |
| 544 | <td class="tableblock halign-right valign-top"><p class="tableblock">1.1 M</p></td> |
| 545 | <td class="tableblock halign-right valign-top"><p class="tableblock">57.7%</p></td> |
| 546 | <td class="tableblock halign-right valign-top"><p class="tableblock">29 bytes</p></td> |
| 547 | <td class="tableblock halign-right valign-top"><p class="tableblock">4 bytes</p></td> |
| 548 | </tr> |
| 549 | <tr> |
| 550 | <td class="tableblock halign-left valign-top"><p class="tableblock">git</p></td> |
| 551 | <td class="tableblock halign-right valign-top"><p class="tableblock">78.7 K</p></td> |
| 552 | <td class="tableblock halign-right valign-top"><p class="tableblock">48.1 K</p></td> |
| 553 | <td class="tableblock halign-right valign-top"><p class="tableblock">61.0%</p></td> |
| 554 | <td class="tableblock halign-right valign-top"><p class="tableblock">50 bytes</p></td> |
| 555 | <td class="tableblock halign-right valign-top"><p class="tableblock">4 bytes</p></td> |
| 556 | </tr> |
| 557 | <tr> |
| 558 | <td class="tableblock halign-left valign-top"><p class="tableblock">git (heads)</p></td> |
| 559 | <td class="tableblock halign-right valign-top"><p class="tableblock">332 b</p></td> |
| 560 | <td class="tableblock halign-right valign-top"><p class="tableblock">269 b</p></td> |
| 561 | <td class="tableblock halign-right valign-top"><p class="tableblock">81.0%</p></td> |
| 562 | <td class="tableblock halign-right valign-top"><p class="tableblock">33 bytes</p></td> |
| 563 | <td class="tableblock halign-right valign-top"><p class="tableblock">0 bytes</p></td> |
| 564 | </tr> |
| 565 | </tbody> |
| 566 | </table> |
| 567 | <div class="paragraph"> |
| 568 | <p>Scan (read 866k refs), by reference name lookup (single ref from 866k |
| 569 | refs), and by SHA-1 lookup (refs with that SHA-1, from 866k refs):</p> |
| 570 | </div> |
| 571 | <table class="tableblock frame-all grid-all stretch"> |
| 572 | <colgroup> |
| 573 | <col style="width: 20%;"/> |
| 574 | <col style="width: 20%;"/> |
| 575 | <col style="width: 20%;"/> |
| 576 | <col style="width: 20%;"/> |
| 577 | <col style="width: 20%;"/> |
| 578 | </colgroup> |
| 579 | <thead> |
| 580 | <tr> |
| 581 | <th class="tableblock halign-left valign-top">format</th> |
| 582 | <th class="tableblock halign-right valign-top">cache</th> |
| 583 | <th class="tableblock halign-right valign-top">scan</th> |
| 584 | <th class="tableblock halign-right valign-top">by name</th> |
| 585 | <th class="tableblock halign-right valign-top">by SHA-1</th> |
| 586 | </tr> |
| 587 | </thead> |
| 588 | <tbody> |
| 589 | <tr> |
| 590 | <td class="tableblock halign-left valign-top"><p class="tableblock">packed-refs</p></td> |
| 591 | <td class="tableblock halign-right valign-top"><p class="tableblock">cold</p></td> |
| 592 | <td class="tableblock halign-right valign-top"><p class="tableblock">402 ms</p></td> |
| 593 | <td class="tableblock halign-right valign-top"><p class="tableblock">409,660.1 usec</p></td> |
| 594 | <td class="tableblock halign-right valign-top"><p class="tableblock">412,535.8 usec</p></td> |
| 595 | </tr> |
| 596 | <tr> |
| 597 | <td class="tableblock halign-left valign-top"><p class="tableblock">packed-refs</p></td> |
| 598 | <td class="tableblock halign-right valign-top"><p class="tableblock">hot</p></td> |
| 599 | <td class="tableblock halign-right valign-top"></td> |
| 600 | <td class="tableblock halign-right valign-top"><p class="tableblock">6,844.6 usec</p></td> |
| 601 | <td class="tableblock halign-right valign-top"><p class="tableblock">20,110.1 usec</p></td> |
| 602 | </tr> |
| 603 | <tr> |
| 604 | <td class="tableblock halign-left valign-top"><p class="tableblock">reftable</p></td> |
| 605 | <td class="tableblock halign-right valign-top"><p class="tableblock">cold</p></td> |
| 606 | <td class="tableblock halign-right valign-top"><p class="tableblock">112 ms</p></td> |
| 607 | <td class="tableblock halign-right valign-top"><p class="tableblock">33.9 usec</p></td> |
| 608 | <td class="tableblock halign-right valign-top"><p class="tableblock">323.2 usec</p></td> |
| 609 | </tr> |
| 610 | <tr> |
| 611 | <td class="tableblock halign-left valign-top"><p class="tableblock">reftable</p></td> |
| 612 | <td class="tableblock halign-right valign-top"><p class="tableblock">hot</p></td> |
| 613 | <td class="tableblock halign-right valign-top"></td> |
| 614 | <td class="tableblock halign-right valign-top"><p class="tableblock">20.2 usec</p></td> |
| 615 | <td class="tableblock halign-right valign-top"><p class="tableblock">320.8 usec</p></td> |
| 616 | </tr> |
| 617 | </tbody> |
| 618 | </table> |
| 619 | <div class="paragraph"> |
| 620 | <p>Space used for 149,932 log entries for 43,061 refs, reflog vs. reftable:</p> |
| 621 | </div> |
| 622 | <table class="tableblock frame-all grid-all stretch"> |
| 623 | <colgroup> |
| 624 | <col style="width: 33.3333%;"/> |
| 625 | <col style="width: 33.3333%;"/> |
| 626 | <col style="width: 33.3334%;"/> |
| 627 | </colgroup> |
| 628 | <thead> |
| 629 | <tr> |
| 630 | <th class="tableblock halign-left valign-top">format</th> |
| 631 | <th class="tableblock halign-right valign-top">size</th> |
| 632 | <th class="tableblock halign-right valign-top">avg entry</th> |
| 633 | </tr> |
| 634 | </thead> |
| 635 | <tbody> |
| 636 | <tr> |
| 637 | <td class="tableblock halign-left valign-top"><p class="tableblock">$GIT_DIR/logs</p></td> |
| 638 | <td class="tableblock halign-right valign-top"><p class="tableblock">173 M</p></td> |
| 639 | <td class="tableblock halign-right valign-top"><p class="tableblock">1209 bytes</p></td> |
| 640 | </tr> |
| 641 | <tr> |
| 642 | <td class="tableblock halign-left valign-top"><p class="tableblock">reftable</p></td> |
| 643 | <td class="tableblock halign-right valign-top"><p class="tableblock">5 M</p></td> |
| 644 | <td class="tableblock halign-right valign-top"><p class="tableblock">37 bytes</p></td> |
| 645 | </tr> |
| 646 | </tbody> |
| 647 | </table> |
| 648 | </div> |
| 649 | </div> |
| 650 | <div class="sect2"> |
| 651 | <h3 id="_details">Details</h3> |
| 652 | <div class="sect3"> |
| 653 | <h4 id="_peeling">Peeling</h4> |
| 654 | <div class="paragraph"> |
| 655 | <p>References stored in a reftable are peeled, a record for an annotated |
| 656 | (or signed) tag records both the tag object, and the object it refers |
| 657 | to. This is analogous to storage in the packed-refs format.</p> |
| 658 | </div> |
| 659 | </div> |
| 660 | <div class="sect3"> |
| 661 | <h4 id="_reference_name_encoding">Reference name encoding</h4> |
| 662 | <div class="paragraph"> |
| 663 | <p>Reference names are an uninterpreted sequence of bytes that must pass |
| 664 | <a href="../git-check-ref-format.html">git-check-ref-format(1)</a> as a valid reference name.</p> |
| 665 | </div> |
| 666 | </div> |
| 667 | <div class="sect3"> |
| 668 | <h4 id="_key_unicity">Key unicity</h4> |
| 669 | <div class="paragraph"> |
| 670 | <p>Each entry must have a unique key; repeated keys are disallowed.</p> |
| 671 | </div> |
| 672 | </div> |
| 673 | <div class="sect3"> |
| 674 | <h4 id="_network_byte_order">Network byte order</h4> |
| 675 | <div class="paragraph"> |
| 676 | <p>All multi-byte, fixed width fields are in network byte order.</p> |
| 677 | </div> |
| 678 | </div> |
| 679 | <div class="sect3"> |
| 680 | <h4 id="_varint_encoding">Varint encoding</h4> |
| 681 | <div class="paragraph"> |
| 682 | <p>Varint encoding is identical to the ofs-delta encoding method used |
| 683 | within pack files.</p> |
| 684 | </div> |
| 685 | <div class="paragraph"> |
| 686 | <p>Decoder works as follows:</p> |
| 687 | </div> |
| 688 | <div class="literalblock"> |
| 689 | <div class="content"> |
| 690 | <pre>val = buf[ptr] & 0x7f |
| 691 | while (buf[ptr] & 0x80) { |
| 692 | ptr++ |
| 693 | val = ((val + 1) << 7) | (buf[ptr] & 0x7f) |
| 694 | }</pre> |
| 695 | </div> |
| 696 | </div> |
| 697 | </div> |
| 698 | <div class="sect3"> |
| 699 | <h4 id="_ordering">Ordering</h4> |
| 700 | <div class="paragraph"> |
| 701 | <p>Blocks are lexicographically ordered by their first reference.</p> |
| 702 | </div> |
| 703 | </div> |
| 704 | <div class="sect3"> |
| 705 | <h4 id="_directoryfile_conflicts">Directory/file conflicts</h4> |
| 706 | <div class="paragraph"> |
| 707 | <p>The reftable format accepts both <code>refs/heads/foo</code> and |
| 708 | <code>refs/heads/foo/bar</code> as distinct references.</p> |
| 709 | </div> |
| 710 | <div class="paragraph"> |
| 711 | <p>This property is useful for retaining log records in reftable, but may |
| 712 | confuse versions of Git using <code>$GIT_DIR/refs</code> directory tree to maintain |
| 713 | references. Users of reftable may choose to continue to reject <code>foo</code> and |
| 714 | <code>foo/bar</code> type conflicts to prevent problems for peers.</p> |
| 715 | </div> |
| 716 | </div> |
| 717 | </div> |
| 718 | <div class="sect2"> |
| 719 | <h3 id="_file_format">File format</h3> |
| 720 | <div class="sect3"> |
| 721 | <h4 id="_structure">Structure</h4> |
| 722 | <div class="paragraph"> |
| 723 | <p>A reftable file has the following high-level structure:</p> |
| 724 | </div> |
| 725 | <div class="literalblock"> |
| 726 | <div class="content"> |
| 727 | <pre>first_block { |
| 728 | header |
| 729 | first_ref_block |
| 730 | } |
| 731 | ref_block* |
| 732 | ref_index* |
| 733 | obj_block* |
| 734 | obj_index* |
| 735 | log_block* |
| 736 | log_index* |
| 737 | footer</pre> |
| 738 | </div> |
| 739 | </div> |
| 740 | <div class="paragraph"> |
| 741 | <p>A log-only file omits the <code>ref_block</code>, <code>ref_index</code>, <code>obj_block</code> and |
| 742 | <code>obj_index</code> sections, containing only the file header and log block:</p> |
| 743 | </div> |
| 744 | <div class="literalblock"> |
| 745 | <div class="content"> |
| 746 | <pre>first_block { |
| 747 | header |
| 748 | } |
| 749 | log_block* |
| 750 | log_index* |
| 751 | footer</pre> |
| 752 | </div> |
| 753 | </div> |
| 754 | <div class="paragraph"> |
| 755 | <p>In a log-only file, the first log block immediately follows the file |
| 756 | header, without padding to block alignment.</p> |
| 757 | </div> |
| 758 | </div> |
| 759 | <div class="sect3"> |
| 760 | <h4 id="_block_size">Block size</h4> |
| 761 | <div class="paragraph"> |
| 762 | <p>The file’s block size is arbitrarily determined by the writer, and does |
| 763 | not have to be a power of 2. The block size must be larger than the |
| 764 | longest reference name or log entry used in the repository, as |
| 765 | references cannot span blocks.</p> |
| 766 | </div> |
| 767 | <div class="paragraph"> |
| 768 | <p>Powers of two that are friendly to the virtual memory system or |
| 769 | filesystem (such as 4k or 8k) are recommended. Larger sizes (64k) can |
| 770 | yield better compression, with a possible increased cost incurred by |
| 771 | readers during access.</p> |
| 772 | </div> |
| 773 | <div class="paragraph"> |
| 774 | <p>The largest block size is <code>16777215</code> bytes (15.99 MiB).</p> |
| 775 | </div> |
| 776 | </div> |
| 777 | <div class="sect3"> |
| 778 | <h4 id="_block_alignment">Block alignment</h4> |
| 779 | <div class="paragraph"> |
| 780 | <p>Writers may choose to align blocks at multiples of the block size by |
| 781 | including <code>padding</code> filled with NUL bytes at the end of a block to round |
| 782 | out to the chosen alignment. When alignment is used, writers must |
| 783 | specify the alignment with the file header’s <code>block_size</code> field.</p> |
| 784 | </div> |
| 785 | <div class="paragraph"> |
| 786 | <p>Block alignment is not required by the file format. Unaligned files must |
| 787 | set <code>block_size = 0</code> in the file header, and omit <code>padding</code>. Unaligned |
| 788 | files with more than one ref block must include the <a href="#Ref-index">ref |
| 789 | index</a> to support fast lookup. Readers must be able to read both aligned |
| 790 | and non-aligned files.</p> |
| 791 | </div> |
| 792 | <div class="paragraph"> |
| 793 | <p>Very small files (e.g. a single ref block) may omit <code>padding</code> and the ref |
| 794 | index to reduce total file size.</p> |
| 795 | </div> |
| 796 | </div> |
| 797 | <div class="sect3"> |
| 798 | <h4 id="_header_version_1">Header (version 1)</h4> |
| 799 | <div class="paragraph"> |
| 800 | <p>A 24-byte header appears at the beginning of the file:</p> |
| 801 | </div> |
| 802 | <div class="literalblock"> |
| 803 | <div class="content"> |
| 804 | <pre>'REFT' |
| 805 | uint8( version_number = 1 ) |
| 806 | uint24( block_size ) |
| 807 | uint64( min_update_index ) |
| 808 | uint64( max_update_index )</pre> |
| 809 | </div> |
| 810 | </div> |
| 811 | <div class="paragraph"> |
| 812 | <p>Aligned files must specify <code>block_size</code> to configure readers with the |
| 813 | expected block alignment. Unaligned files must set <code>block_size = 0</code>.</p> |
| 814 | </div> |
| 815 | <div class="paragraph"> |
| 816 | <p>The <code>min_update_index</code> and <code>max_update_index</code> describe bounds for the |
| 817 | <code>update_index</code> field of all log records in this file. When reftables are |
| 818 | used in a stack for <a href="#Update-transactions">transactions</a>, these |
| 819 | fields can order the files such that the prior file’s |
| 820 | <code>max_update_index + 1</code> is the next file’s <code>min_update_index</code>.</p> |
| 821 | </div> |
| 822 | </div> |
| 823 | <div class="sect3"> |
| 824 | <h4 id="_header_version_2">Header (version 2)</h4> |
| 825 | <div class="paragraph"> |
| 826 | <p>A 28-byte header appears at the beginning of the file:</p> |
| 827 | </div> |
| 828 | <div class="literalblock"> |
| 829 | <div class="content"> |
| 830 | <pre>'REFT' |
| 831 | uint8( version_number = 2 ) |
| 832 | uint24( block_size ) |
| 833 | uint64( min_update_index ) |
| 834 | uint64( max_update_index ) |
| 835 | uint32( hash_id )</pre> |
| 836 | </div> |
| 837 | </div> |
| 838 | <div class="paragraph"> |
| 839 | <p>The header is identical to <code>version_number=1</code>, with the 4-byte hash ID |
| 840 | ("sha1" for SHA1 and "s256" for SHA-256) appended to the header.</p> |
| 841 | </div> |
| 842 | <div class="paragraph"> |
| 843 | <p>For maximum backward compatibility, it is recommended to use version 1 when |
| 844 | writing SHA1 reftables.</p> |
| 845 | </div> |
| 846 | </div> |
| 847 | <div class="sect3"> |
| 848 | <h4 id="_first_ref_block">First ref block</h4> |
| 849 | <div class="paragraph"> |
| 850 | <p>The first ref block shares the same block as the file header, and is 24 |
| 851 | bytes smaller than all other blocks in the file. The first block |
| 852 | immediately begins after the file header, at position 24.</p> |
| 853 | </div> |
| 854 | <div class="paragraph"> |
| 855 | <p>If the first block is a log block (a log-only file), its block header |
| 856 | begins immediately at position 24.</p> |
| 857 | </div> |
| 858 | </div> |
| 859 | <div class="sect3"> |
| 860 | <h4 id="_ref_block_format">Ref block format</h4> |
| 861 | <div class="paragraph"> |
| 862 | <p>A ref block is written as:</p> |
| 863 | </div> |
| 864 | <div class="literalblock"> |
| 865 | <div class="content"> |
| 866 | <pre>'r' |
| 867 | uint24( block_len ) |
| 868 | ref_record+ |
| 869 | uint24( restart_offset )+ |
| 870 | uint16( restart_count ) |
| 871 | |
| 872 | padding?</pre> |
| 873 | </div> |
| 874 | </div> |
| 875 | <div class="paragraph"> |
| 876 | <p>Blocks begin with <code>block_type = 'r'</code> and a 3-byte <code>block_len</code> which |
| 877 | encodes the number of bytes in the block up to, but not including the |
| 878 | optional <code>padding</code>. This is always less than or equal to the file’s |
| 879 | block size. In the first ref block, <code>block_len</code> includes 24 bytes for |
| 880 | the file header.</p> |
| 881 | </div> |
| 882 | <div class="paragraph"> |
| 883 | <p>The 2-byte <code>restart_count</code> stores the number of entries in the |
| 884 | <code>restart_offset</code> list, which must not be empty. Readers can use |
| 885 | <code>restart_count</code> to binary search between restarts before starting a |
| 886 | linear scan.</p> |
| 887 | </div> |
| 888 | <div class="paragraph"> |
| 889 | <p>Exactly <code>restart_count</code> 3-byte <code>restart_offset</code> values precede the |
| 890 | <code>restart_count</code>. Offsets are relative to the start of the block and |
| 891 | refer to the first byte of any <code>ref_record</code> whose name has not been |
| 892 | prefix compressed. Entries in the <code>restart_offset</code> list must be sorted, |
| 893 | ascending. Readers can start linear scans from any of these records.</p> |
| 894 | </div> |
| 895 | <div class="paragraph"> |
| 896 | <p>A variable number of <code>ref_record</code> fill the middle of the block, |
| 897 | describing reference names and values. The format is described below.</p> |
| 898 | </div> |
| 899 | <div class="paragraph"> |
| 900 | <p>As the first ref block shares the first file block with the file header, |
| 901 | all <code>restart_offset</code> in the first block are relative to the start of the |
| 902 | file (position 0), and include the file header. This forces the first |
| 903 | <code>restart_offset</code> to be <code>28</code>.</p> |
| 904 | </div> |
| 905 | <div class="sect4"> |
| 906 | <h5 id="_ref_record">ref record</h5> |
| 907 | <div class="paragraph"> |
| 908 | <p>A <code>ref_record</code> describes a single reference, storing both the name and |
| 909 | its value(s). Records are formatted as:</p> |
| 910 | </div> |
| 911 | <div class="literalblock"> |
| 912 | <div class="content"> |
| 913 | <pre>varint( prefix_length ) |
| 914 | varint( (suffix_length << 3) | value_type ) |
| 915 | suffix |
| 916 | varint( update_index_delta ) |
| 917 | value?</pre> |
| 918 | </div> |
| 919 | </div> |
| 920 | <div class="paragraph"> |
| 921 | <p>The <code>prefix_length</code> field specifies how many leading bytes of the prior |
| 922 | reference record’s name should be copied to obtain this reference’s |
| 923 | name. This must be 0 for the first reference in any block, and also must |
| 924 | be 0 for any <code>ref_record</code> whose offset is listed in the <code>restart_offset</code> |
| 925 | table at the end of the block.</p> |
| 926 | </div> |
| 927 | <div class="paragraph"> |
| 928 | <p>Recovering a reference name from any <code>ref_record</code> is a simple concat:</p> |
| 929 | </div> |
| 930 | <div class="literalblock"> |
| 931 | <div class="content"> |
| 932 | <pre>this_name = prior_name[0..prefix_length] + suffix</pre> |
| 933 | </div> |
| 934 | </div> |
| 935 | <div class="paragraph"> |
| 936 | <p>The <code>suffix_length</code> value provides the number of bytes available in |
| 937 | <code>suffix</code> to copy from <code>suffix</code> to complete the reference name.</p> |
| 938 | </div> |
| 939 | <div class="paragraph"> |
| 940 | <p>The <code>update_index</code> that last modified the reference can be obtained by |
| 941 | adding <code>update_index_delta</code> to the <code>min_update_index</code> from the file |
| 942 | header: <code>min_update_index + update_index_delta</code>.</p> |
| 943 | </div> |
| 944 | <div class="paragraph"> |
| 945 | <p>The <code>value</code> follows. Its format is determined by <code>value_type</code>, one of |
| 946 | the following:</p> |
| 947 | </div> |
| 948 | <div class="ulist"> |
| 949 | <ul> |
| 950 | <li> |
| 951 | <p><code>0x0</code>: deletion; no value data (see transactions, below)</p> |
| 952 | </li> |
| 953 | <li> |
| 954 | <p><code>0x1</code>: one object name; value of the ref</p> |
| 955 | </li> |
| 956 | <li> |
| 957 | <p><code>0x2</code>: two object names; value of the ref, peeled target</p> |
| 958 | </li> |
| 959 | <li> |
| 960 | <p><code>0x3</code>: symbolic reference: <code>varint( target_len ) target</code></p> |
| 961 | </li> |
| 962 | </ul> |
| 963 | </div> |
| 964 | <div class="paragraph"> |
| 965 | <p>Symbolic references use <code>0x3</code>, followed by the complete name of the |
| 966 | reference target. No compression is applied to the target name.</p> |
| 967 | </div> |
| 968 | <div class="paragraph"> |
| 969 | <p>Types <code>0x4..0x7</code> are reserved for future use.</p> |
| 970 | </div> |
| 971 | </div> |
| 972 | </div> |
| 973 | <div class="sect3"> |
| 974 | <h4 id="_ref_index">Ref index</h4> |
| 975 | <div class="paragraph"> |
| 976 | <p>The ref index stores the name of the last reference from every ref block |
| 977 | in the file, enabling reduced disk seeks for lookups. Any reference can |
| 978 | be found by searching the index, identifying the containing block, and |
| 979 | searching within that block.</p> |
| 980 | </div> |
| 981 | <div class="paragraph"> |
| 982 | <p>The index may be organized into a multi-level index, where the 1st level |
| 983 | index block points to additional ref index blocks (2nd level), which may |
| 984 | in turn point to either additional index blocks (e.g. 3rd level) or ref |
| 985 | blocks (leaf level). Disk reads required to access a ref go up with |
| 986 | higher index levels. Multi-level indexes may be required to ensure no |
| 987 | single index block exceeds the file format’s max block size of |
| 988 | <code>16777215</code> bytes (15.99 MiB). To achieve constant O(1) disk seeks for |
| 989 | lookups the index must be a single level, which is permitted to exceed |
| 990 | the file’s configured block size, but not the format’s max block size of |
| 991 | 15.99 MiB.</p> |
| 992 | </div> |
| 993 | <div class="paragraph"> |
| 994 | <p>If present, the ref index block(s) appears after the last ref block.</p> |
| 995 | </div> |
| 996 | <div class="paragraph"> |
| 997 | <p>If there are at least 4 ref blocks, a ref index block should be written |
| 998 | to improve lookup times. Cold reads using the index require 2 disk reads |
| 999 | (read index, read block), and binary searching < 4 blocks also requires |
| 1000 | ⇐ 2 reads. Omitting the index block from smaller files saves space.</p> |
| 1001 | </div> |
| 1002 | <div class="paragraph"> |
| 1003 | <p>If the file is unaligned and contains more than one ref block, the ref |
| 1004 | index must be written.</p> |
| 1005 | </div> |
| 1006 | <div class="paragraph"> |
| 1007 | <p>Index block format:</p> |
| 1008 | </div> |
| 1009 | <div class="literalblock"> |
| 1010 | <div class="content"> |
| 1011 | <pre>'i' |
| 1012 | uint24( block_len ) |
| 1013 | index_record+ |
| 1014 | uint24( restart_offset )+ |
| 1015 | uint16( restart_count ) |
| 1016 | |
| 1017 | padding?</pre> |
| 1018 | </div> |
| 1019 | </div> |
| 1020 | <div class="paragraph"> |
| 1021 | <p>The index blocks begin with <code>block_type = 'i'</code> and a 3-byte <code>block_len</code> |
| 1022 | which encodes the number of bytes in the block, up to but not including |
| 1023 | the optional <code>padding</code>.</p> |
| 1024 | </div> |
| 1025 | <div class="paragraph"> |
| 1026 | <p>The <code>restart_offset</code> and <code>restart_count</code> fields are identical in format, |
| 1027 | meaning and usage as in ref blocks.</p> |
| 1028 | </div> |
| 1029 | <div class="paragraph"> |
| 1030 | <p>To reduce the number of reads required for random access in very large |
| 1031 | files the index block may be larger than other blocks. However, readers |
| 1032 | must hold the entire index in memory to benefit from this, so it’s a |
| 1033 | time-space tradeoff in both file size and reader memory.</p> |
| 1034 | </div> |
| 1035 | <div class="paragraph"> |
| 1036 | <p>Increasing the file’s block size decreases the index size. Alternatively |
| 1037 | a multi-level index may be used, keeping index blocks within the file’s |
| 1038 | block size, but increasing the number of blocks that need to be |
| 1039 | accessed.</p> |
| 1040 | </div> |
| 1041 | <div class="sect4"> |
| 1042 | <h5 id="_index_record">index record</h5> |
| 1043 | <div class="paragraph"> |
| 1044 | <p>An index record describes the last entry in another block. Index records |
| 1045 | are written as:</p> |
| 1046 | </div> |
| 1047 | <div class="literalblock"> |
| 1048 | <div class="content"> |
| 1049 | <pre>varint( prefix_length ) |
| 1050 | varint( (suffix_length << 3) | 0 ) |
| 1051 | suffix |
| 1052 | varint( block_position )</pre> |
| 1053 | </div> |
| 1054 | </div> |
| 1055 | <div class="paragraph"> |
| 1056 | <p>Index records use prefix compression exactly like <code>ref_record</code>.</p> |
| 1057 | </div> |
| 1058 | <div class="paragraph"> |
| 1059 | <p>Index records store <code>block_position</code> after the suffix, specifying the |
| 1060 | absolute position in bytes (from the start of the file) of the block |
| 1061 | that ends with this reference. Readers can seek to <code>block_position</code> to |
| 1062 | begin reading the block header.</p> |
| 1063 | </div> |
| 1064 | <div class="paragraph"> |
| 1065 | <p>Readers must examine the block header at <code>block_position</code> to determine |
| 1066 | if the next block is another level index block, or the leaf-level ref |
| 1067 | block.</p> |
| 1068 | </div> |
| 1069 | </div> |
| 1070 | <div class="sect4"> |
| 1071 | <h5 id="_reading_the_index">Reading the index</h5> |
| 1072 | <div class="paragraph"> |
| 1073 | <p>Readers loading the ref index must first read the footer (below) to |
| 1074 | obtain <code>ref_index_position</code>. If not present, the position will be 0. The |
| 1075 | <code>ref_index_position</code> is for the 1st level root of the ref index.</p> |
| 1076 | </div> |
| 1077 | </div> |
| 1078 | </div> |
| 1079 | <div class="sect3"> |
| 1080 | <h4 id="_obj_block_format">Obj block format</h4> |
| 1081 | <div class="paragraph"> |
| 1082 | <p>Object blocks are optional. Writers may choose to omit object blocks, |
| 1083 | especially if readers will not use the object name to ref mapping.</p> |
| 1084 | </div> |
| 1085 | <div class="paragraph"> |
| 1086 | <p>Object blocks use unique, abbreviated 2-31 byte object name keys, mapping to |
| 1087 | ref blocks containing references pointing to that object directly, or as |
| 1088 | the peeled value of an annotated tag. Like ref blocks, object blocks use |
| 1089 | the file’s standard block size. The abbreviation length is available in |
| 1090 | the footer as <code>obj_id_len</code>.</p> |
| 1091 | </div> |
| 1092 | <div class="paragraph"> |
| 1093 | <p>To save space in small files, object blocks may be omitted if the ref |
| 1094 | index is not present, as brute force search will only need to read a few |
| 1095 | ref blocks. When missing, readers should brute force a linear search of |
| 1096 | all references to lookup by object name.</p> |
| 1097 | </div> |
| 1098 | <div class="paragraph"> |
| 1099 | <p>An object block is written as:</p> |
| 1100 | </div> |
| 1101 | <div class="literalblock"> |
| 1102 | <div class="content"> |
| 1103 | <pre>'o' |
| 1104 | uint24( block_len ) |
| 1105 | obj_record+ |
| 1106 | uint24( restart_offset )+ |
| 1107 | uint16( restart_count ) |
| 1108 | |
| 1109 | padding?</pre> |
| 1110 | </div> |
| 1111 | </div> |
| 1112 | <div class="paragraph"> |
| 1113 | <p>Fields are identical to ref block. Binary search using the restart table |
| 1114 | works the same as in reference blocks.</p> |
| 1115 | </div> |
| 1116 | <div class="paragraph"> |
| 1117 | <p>Because object names are abbreviated by writers to the shortest unique |
| 1118 | abbreviation within the reftable, obj key lengths have a variable length. Their |
| 1119 | length must be at least 2 bytes. Readers must compare only for common prefix |
| 1120 | match within an obj block or obj index.</p> |
| 1121 | </div> |
| 1122 | <div class="sect4"> |
| 1123 | <h5 id="_obj_record">obj record</h5> |
| 1124 | <div class="paragraph"> |
| 1125 | <p>An <code>obj_record</code> describes a single object abbreviation, and the blocks |
| 1126 | containing references using that unique abbreviation:</p> |
| 1127 | </div> |
| 1128 | <div class="literalblock"> |
| 1129 | <div class="content"> |
| 1130 | <pre>varint( prefix_length ) |
| 1131 | varint( (suffix_length << 3) | cnt_3 ) |
| 1132 | suffix |
| 1133 | varint( cnt_large )? |
| 1134 | varint( position_delta )*</pre> |
| 1135 | </div> |
| 1136 | </div> |
| 1137 | <div class="paragraph"> |
| 1138 | <p>Like in reference blocks, abbreviations are prefix compressed within an |
| 1139 | obj block. On large reftables with many unique objects, higher block |
| 1140 | sizes (64k), and higher restart interval (128), a <code>prefix_length</code> of 2 |
| 1141 | or 3 and <code>suffix_length</code> of 3 may be common in obj records (unique |
| 1142 | abbreviation of 5-6 raw bytes, 10-12 hex digits).</p> |
| 1143 | </div> |
| 1144 | <div class="paragraph"> |
| 1145 | <p>Each record contains <code>position_count</code> number of positions for matching |
| 1146 | ref blocks. For 1-7 positions the count is stored in <code>cnt_3</code>. When |
| 1147 | <code>cnt_3 = 0</code> the actual count follows in a varint, <code>cnt_large</code>.</p> |
| 1148 | </div> |
| 1149 | <div class="paragraph"> |
| 1150 | <p>The use of <code>cnt_3</code> bets most objects are pointed to by only a single |
| 1151 | reference, some may be pointed to by a couple of references, and very |
| 1152 | few (if any) are pointed to by more than 7 references.</p> |
| 1153 | </div> |
| 1154 | <div class="paragraph"> |
| 1155 | <p>A special case exists when <code>cnt_3 = 0</code> and <code>cnt_large = 0</code>: there are no |
| 1156 | <code>position_delta</code>, but at least one reference starts with this |
| 1157 | abbreviation. A reader that needs exact reference names must scan all |
| 1158 | references to find which specific references have the desired object. |
| 1159 | Writers should use this format when the <code>position_delta</code> list would have |
| 1160 | overflowed the file’s block size due to a high number of references |
| 1161 | pointing to the same object.</p> |
| 1162 | </div> |
| 1163 | <div class="paragraph"> |
| 1164 | <p>The first <code>position_delta</code> is the position from the start of the file. |
| 1165 | Additional <code>position_delta</code> entries are sorted ascending and relative to |
| 1166 | the prior entry, e.g. a reader would perform:</p> |
| 1167 | </div> |
| 1168 | <div class="literalblock"> |
| 1169 | <div class="content"> |
| 1170 | <pre>pos = position_delta[0] |
| 1171 | prior = pos |
| 1172 | for (j = 1; j < position_count; j++) { |
| 1173 | pos = prior + position_delta[j] |
| 1174 | prior = pos |
| 1175 | }</pre> |
| 1176 | </div> |
| 1177 | </div> |
| 1178 | <div class="paragraph"> |
| 1179 | <p>With a position in hand, a reader must linearly scan the ref block, |
| 1180 | starting from the first <code>ref_record</code>, testing each reference’s object names |
| 1181 | (for <code>value_type = 0x1</code> or <code>0x2</code>) for full equality. Faster searching by |
| 1182 | object name within a single ref block is not supported by the reftable format. |
| 1183 | Smaller block sizes reduce the number of candidates this step must |
| 1184 | consider.</p> |
| 1185 | </div> |
| 1186 | </div> |
| 1187 | </div> |
| 1188 | <div class="sect3"> |
| 1189 | <h4 id="_obj_index">Obj index</h4> |
| 1190 | <div class="paragraph"> |
| 1191 | <p>The obj index stores the abbreviation from the last entry for every obj |
| 1192 | block in the file, enabling reduced disk seeks for all lookups. It is |
| 1193 | formatted exactly the same as the ref index, but refers to obj blocks.</p> |
| 1194 | </div> |
| 1195 | <div class="paragraph"> |
| 1196 | <p>The obj index should be present if obj blocks are present, as obj blocks |
| 1197 | should only be written in larger files.</p> |
| 1198 | </div> |
| 1199 | <div class="paragraph"> |
| 1200 | <p>Readers loading the obj index must first read the footer (below) to |
| 1201 | obtain <code>obj_index_position</code>. If not present, the position will be 0.</p> |
| 1202 | </div> |
| 1203 | </div> |
| 1204 | <div class="sect3"> |
| 1205 | <h4 id="_log_block_format">Log block format</h4> |
| 1206 | <div class="paragraph"> |
| 1207 | <p>Unlike ref and obj blocks, log blocks are always unaligned.</p> |
| 1208 | </div> |
| 1209 | <div class="paragraph"> |
| 1210 | <p>Log blocks are variable in size, and do not match the <code>block_size</code> |
| 1211 | specified in the file header or footer. Writers should choose an |
| 1212 | appropriate buffer size to prepare a log block for deflation, such as |
| 1213 | <code>2 * block_size</code>.</p> |
| 1214 | </div> |
| 1215 | <div class="paragraph"> |
| 1216 | <p>A log block is written as:</p> |
| 1217 | </div> |
| 1218 | <div class="literalblock"> |
| 1219 | <div class="content"> |
| 1220 | <pre>'g' |
| 1221 | uint24( block_len ) |
| 1222 | zlib_deflate { |
| 1223 | log_record+ |
| 1224 | uint24( restart_offset )+ |
| 1225 | uint16( restart_count ) |
| 1226 | }</pre> |
| 1227 | </div> |
| 1228 | </div> |
| 1229 | <div class="paragraph"> |
| 1230 | <p>Log blocks look similar to ref blocks, except <code>block_type = 'g'</code>.</p> |
| 1231 | </div> |
| 1232 | <div class="paragraph"> |
| 1233 | <p>The 4-byte block header is followed by the deflated block contents using |
| 1234 | zlib deflate. The <code>block_len</code> in the header is the inflated size |
| 1235 | (including 4-byte block header), and should be used by readers to |
| 1236 | preallocate the inflation output buffer. A log block’s <code>block_len</code> may |
| 1237 | exceed the file’s block size.</p> |
| 1238 | </div> |
| 1239 | <div class="paragraph"> |
| 1240 | <p>Offsets within the log block (e.g. <code>restart_offset</code>) still include the |
| 1241 | 4-byte header. Readers may prefer prefixing the inflation output buffer |
| 1242 | with the 4-byte header.</p> |
| 1243 | </div> |
| 1244 | <div class="paragraph"> |
| 1245 | <p>Within the deflate container, a variable number of <code>log_record</code> describe |
| 1246 | reference changes. The log record format is described below. See ref |
| 1247 | block format (above) for a description of <code>restart_offset</code> and |
| 1248 | <code>restart_count</code>.</p> |
| 1249 | </div> |
| 1250 | <div class="paragraph"> |
| 1251 | <p>Because log blocks have no alignment or padding between blocks, readers |
| 1252 | must keep track of the bytes consumed by the inflater to know where the |
| 1253 | next log block begins.</p> |
| 1254 | </div> |
| 1255 | <div class="sect4"> |
| 1256 | <h5 id="_log_record">log record</h5> |
| 1257 | <div class="paragraph"> |
| 1258 | <p>Log record keys are structured as:</p> |
| 1259 | </div> |
| 1260 | <div class="literalblock"> |
| 1261 | <div class="content"> |
| 1262 | <pre>ref_name '\0' reverse_int64( update_index )</pre> |
| 1263 | </div> |
| 1264 | </div> |
| 1265 | <div class="paragraph"> |
| 1266 | <p>where <code>update_index</code> is the unique transaction identifier. The |
| 1267 | <code>update_index</code> field must be unique within the scope of a <code>ref_name</code>. |
| 1268 | See the update transactions section below for further details.</p> |
| 1269 | </div> |
| 1270 | <div class="paragraph"> |
| 1271 | <p>The <code>reverse_int64</code> function inverses the value so lexicographical |
| 1272 | ordering the network byte order encoding sorts the more recent records |
| 1273 | with higher <code>update_index</code> values first:</p> |
| 1274 | </div> |
| 1275 | <div class="literalblock"> |
| 1276 | <div class="content"> |
| 1277 | <pre>reverse_int64(int64 t) { |
| 1278 | return 0xffffffffffffffff - t; |
| 1279 | }</pre> |
| 1280 | </div> |
| 1281 | </div> |
| 1282 | <div class="paragraph"> |
| 1283 | <p>Log records have a similar starting structure to ref and index records, |
| 1284 | utilizing the same prefix compression scheme applied to the log record |
| 1285 | key described above.</p> |
| 1286 | </div> |
| 1287 | <div class="literalblock"> |
| 1288 | <div class="content"> |
| 1289 | <pre> varint( prefix_length ) |
| 1290 | varint( (suffix_length << 3) | log_type ) |
| 1291 | suffix |
| 1292 | log_data { |
| 1293 | old_id |
| 1294 | new_id |
| 1295 | varint( name_length ) name |
| 1296 | varint( email_length ) email |
| 1297 | varint( time_seconds ) |
| 1298 | sint16( tz_offset ) |
| 1299 | varint( message_length ) message |
| 1300 | }?</pre> |
| 1301 | </div> |
| 1302 | </div> |
| 1303 | <div class="paragraph"> |
| 1304 | <p>Log record entries use <code>log_type</code> to indicate what follows:</p> |
| 1305 | </div> |
| 1306 | <div class="ulist"> |
| 1307 | <ul> |
| 1308 | <li> |
| 1309 | <p><code>0x0</code>: deletion; no log data.</p> |
| 1310 | </li> |
| 1311 | <li> |
| 1312 | <p><code>0x1</code>: standard git reflog data using <code>log_data</code> above.</p> |
| 1313 | </li> |
| 1314 | </ul> |
| 1315 | </div> |
| 1316 | <div class="paragraph"> |
| 1317 | <p>The <code>log_type = 0x0</code> is mostly useful for <code>git stash drop</code>, removing an |
| 1318 | entry from the reflog of <code>refs/stash</code> in a transaction file (below), |
| 1319 | without needing to rewrite larger files. Readers reading a stack of |
| 1320 | reflogs must treat this as a deletion.</p> |
| 1321 | </div> |
| 1322 | <div class="paragraph"> |
| 1323 | <p>For <code>log_type = 0x1</code>, the <code>log_data</code> section follows |
| 1324 | <a href="../git-update-ref.html">git-update-ref(1)</a> logging and includes:</p> |
| 1325 | </div> |
| 1326 | <div class="ulist"> |
| 1327 | <ul> |
| 1328 | <li> |
| 1329 | <p>two object names (old id, new id)</p> |
| 1330 | </li> |
| 1331 | <li> |
| 1332 | <p>varint string of committer’s name</p> |
| 1333 | </li> |
| 1334 | <li> |
| 1335 | <p>varint string of committer’s email</p> |
| 1336 | </li> |
| 1337 | <li> |
| 1338 | <p>varint time in seconds since epoch (Jan 1, 1970)</p> |
| 1339 | </li> |
| 1340 | <li> |
| 1341 | <p>2-byte timezone offset in minutes (signed)</p> |
| 1342 | </li> |
| 1343 | <li> |
| 1344 | <p>varint string of message</p> |
| 1345 | </li> |
| 1346 | </ul> |
| 1347 | </div> |
| 1348 | <div class="paragraph"> |
| 1349 | <p><code>tz_offset</code> is the absolute number of minutes from GMT the committer was |
| 1350 | at the time of the update. For example <code>GMT-0800</code> is encoded in reftable |
| 1351 | as <code>sint16(-480)</code> and <code>GMT+0230</code> is <code>sint16(150)</code>.</p> |
| 1352 | </div> |
| 1353 | <div class="paragraph"> |
| 1354 | <p>The committer email does not contain <code><</code> or <code>></code>, it’s the value normally |
| 1355 | found between the <code><></code> in a git commit object header.</p> |
| 1356 | </div> |
| 1357 | <div class="paragraph"> |
| 1358 | <p>The <code>message_length</code> may be 0, in which case there was no message |
| 1359 | supplied for the update.</p> |
| 1360 | </div> |
| 1361 | <div class="paragraph"> |
| 1362 | <p>Contrary to traditional reflog (which is a file), renames are encoded as |
| 1363 | a combination of ref deletion and ref creation. A deletion is a log |
| 1364 | record with a zero new_id, and a creation is a log record with a zero old_id.</p> |
| 1365 | </div> |
| 1366 | </div> |
| 1367 | <div class="sect4"> |
| 1368 | <h5 id="_reading_the_log">Reading the log</h5> |
| 1369 | <div class="paragraph"> |
| 1370 | <p>Readers accessing the log must first read the footer (below) to |
| 1371 | determine the <code>log_position</code>. The first block of the log begins at |
| 1372 | <code>log_position</code> bytes since the start of the file. The <code>log_position</code> is |
| 1373 | not block aligned.</p> |
| 1374 | </div> |
| 1375 | </div> |
| 1376 | <div class="sect4"> |
| 1377 | <h5 id="_importing_logs">Importing logs</h5> |
| 1378 | <div class="paragraph"> |
| 1379 | <p>When importing from <code>$GIT_DIR/logs</code> writers should globally order all |
| 1380 | log records roughly by timestamp while preserving file order, and assign |
| 1381 | unique, increasing <code>update_index</code> values for each log line. Newer log |
| 1382 | records get higher <code>update_index</code> values.</p> |
| 1383 | </div> |
| 1384 | <div class="paragraph"> |
| 1385 | <p>Although an import may write only a single reftable file, the reftable |
| 1386 | file must span many unique <code>update_index</code>, as each log line requires its |
| 1387 | own <code>update_index</code> to preserve semantics.</p> |
| 1388 | </div> |
| 1389 | </div> |
| 1390 | </div> |
| 1391 | <div class="sect3"> |
| 1392 | <h4 id="_log_index">Log index</h4> |
| 1393 | <div class="paragraph"> |
| 1394 | <p>The log index stores the log key |
| 1395 | (<code>refname \0 reverse_int64(update_index)</code>) for the last log record of |
| 1396 | every log block in the file, supporting bounded-time lookup.</p> |
| 1397 | </div> |
| 1398 | <div class="paragraph"> |
| 1399 | <p>A log index block must be written if 2 or more log blocks are written to |
| 1400 | the file. If present, the log index appears after the last log block. |
| 1401 | There is no padding used to align the log index to block alignment.</p> |
| 1402 | </div> |
| 1403 | <div class="paragraph"> |
| 1404 | <p>Log index format is identical to ref index, except the keys are 9 bytes |
| 1405 | longer to include <code>'\0'</code> and the 8-byte <code>reverse_int64(update_index)</code>. |
| 1406 | Records use <code>block_position</code> to refer to the start of a log block.</p> |
| 1407 | </div> |
| 1408 | <div class="sect4"> |
| 1409 | <h5 id="_reading_the_index_2">Reading the index</h5> |
| 1410 | <div class="paragraph"> |
| 1411 | <p>Readers loading the log index must first read the footer (below) to |
| 1412 | obtain <code>log_index_position</code>. If not present, the position will be 0.</p> |
| 1413 | </div> |
| 1414 | </div> |
| 1415 | </div> |
| 1416 | <div class="sect3"> |
| 1417 | <h4 id="_footer">Footer</h4> |
| 1418 | <div class="paragraph"> |
| 1419 | <p>After the last block of the file, a file footer is written. It begins |
| 1420 | like the file header, but is extended with additional data.</p> |
| 1421 | </div> |
| 1422 | <div class="literalblock"> |
| 1423 | <div class="content"> |
| 1424 | <pre> HEADER |
| 1425 | |
| 1426 | uint64( ref_index_position ) |
| 1427 | uint64( (obj_position << 5) | obj_id_len ) |
| 1428 | uint64( obj_index_position ) |
| 1429 | |
| 1430 | uint64( log_position ) |
| 1431 | uint64( log_index_position ) |
| 1432 | |
| 1433 | uint32( CRC-32 of above )</pre> |
| 1434 | </div> |
| 1435 | </div> |
| 1436 | <div class="paragraph"> |
| 1437 | <p>If a section is missing (e.g. ref index) the corresponding position |
| 1438 | field (e.g. <code>ref_index_position</code>) will be 0.</p> |
| 1439 | </div> |
| 1440 | <div class="ulist"> |
| 1441 | <ul> |
| 1442 | <li> |
| 1443 | <p><code>obj_position</code>: byte position for the first obj block.</p> |
| 1444 | </li> |
| 1445 | <li> |
| 1446 | <p><code>obj_id_len</code>: number of bytes used to abbreviate object names in |
| 1447 | obj blocks.</p> |
| 1448 | </li> |
| 1449 | <li> |
| 1450 | <p><code>log_position</code>: byte position for the first log block.</p> |
| 1451 | </li> |
| 1452 | <li> |
| 1453 | <p><code>ref_index_position</code>: byte position for the start of the ref index.</p> |
| 1454 | </li> |
| 1455 | <li> |
| 1456 | <p><code>obj_index_position</code>: byte position for the start of the obj index.</p> |
| 1457 | </li> |
| 1458 | <li> |
| 1459 | <p><code>log_index_position</code>: byte position for the start of the log index.</p> |
| 1460 | </li> |
| 1461 | </ul> |
| 1462 | </div> |
| 1463 | <div class="paragraph"> |
| 1464 | <p>The size of the footer is 68 bytes for version 1, and 72 bytes for |
| 1465 | version 2.</p> |
| 1466 | </div> |
| 1467 | <div class="sect4"> |
| 1468 | <h5 id="_reading_the_footer">Reading the footer</h5> |
| 1469 | <div class="paragraph"> |
| 1470 | <p>Readers must first read the file start to determine the version |
| 1471 | number. Then they seek to <code>file_length - FOOTER_LENGTH</code> to access the |
| 1472 | footer. A trusted external source (such as <code>stat(2)</code>) is necessary to |
| 1473 | obtain <code>file_length</code>. When reading the footer, readers must verify:</p> |
| 1474 | </div> |
| 1475 | <div class="ulist"> |
| 1476 | <ul> |
| 1477 | <li> |
| 1478 | <p>4-byte magic is correct</p> |
| 1479 | </li> |
| 1480 | <li> |
| 1481 | <p>1-byte version number is recognized</p> |
| 1482 | </li> |
| 1483 | <li> |
| 1484 | <p>4-byte CRC-32 matches the other 64 bytes (including magic, and |
| 1485 | version)</p> |
| 1486 | </li> |
| 1487 | </ul> |
| 1488 | </div> |
| 1489 | <div class="paragraph"> |
| 1490 | <p>Once verified, the other fields of the footer can be accessed.</p> |
| 1491 | </div> |
| 1492 | </div> |
| 1493 | <div class="sect4"> |
| 1494 | <h5 id="_empty_tables">Empty tables</h5> |
| 1495 | <div class="paragraph"> |
| 1496 | <p>A reftable may be empty. In this case, the file starts with a header |
| 1497 | and is immediately followed by a footer.</p> |
| 1498 | </div> |
| 1499 | </div> |
| 1500 | </div> |
| 1501 | <div class="sect3"> |
| 1502 | <h4 id="_binary_search">Binary search</h4> |
| 1503 | <div class="paragraph"> |
| 1504 | <p>Binary search within a block is supported by the <code>restart_offset</code> fields |
| 1505 | at the end of the block. Readers can binary search through the restart |
| 1506 | table to locate between which two restart points the sought reference or |
| 1507 | key should appear.</p> |
| 1508 | </div> |
| 1509 | <div class="paragraph"> |
| 1510 | <p>Each record identified by a <code>restart_offset</code> stores the complete key in |
| 1511 | the <code>suffix</code> field of the record, making the compare operation during |
| 1512 | binary search straightforward.</p> |
| 1513 | </div> |
| 1514 | <div class="paragraph"> |
| 1515 | <p>Once a restart point lexicographically before the sought reference has |
| 1516 | been identified, readers can linearly scan through the following record |
| 1517 | entries to locate the sought record, terminating if the current record |
| 1518 | sorts after (and therefore the sought key is not present).</p> |
| 1519 | </div> |
| 1520 | <div class="sect4"> |
| 1521 | <h5 id="_restart_point_selection">Restart point selection</h5> |
| 1522 | <div class="paragraph"> |
| 1523 | <p>Writers determine the restart points at file creation. The process is |
| 1524 | arbitrary, but every 16 or 64 records is recommended. Every 16 may be |
| 1525 | more suitable for smaller block sizes (4k or 8k), every 64 for larger |
| 1526 | block sizes (64k).</p> |
| 1527 | </div> |
| 1528 | <div class="paragraph"> |
| 1529 | <p>More frequent restart points reduces prefix compression and increases |
| 1530 | space consumed by the restart table, both of which increase file size.</p> |
| 1531 | </div> |
| 1532 | <div class="paragraph"> |
| 1533 | <p>Less frequent restart points makes prefix compression more effective, |
| 1534 | decreasing overall file size, with increased penalties for readers |
| 1535 | walking through more records after the binary search step.</p> |
| 1536 | </div> |
| 1537 | <div class="paragraph"> |
| 1538 | <p>A maximum of <code>65535</code> restart points per block is supported.</p> |
| 1539 | </div> |
| 1540 | </div> |
| 1541 | </div> |
| 1542 | </div> |
| 1543 | <div class="sect2"> |
| 1544 | <h3 id="_considerations">Considerations</h3> |
| 1545 | <div class="sect3"> |
| 1546 | <h4 id="_lightweight_refs_dominate">Lightweight refs dominate</h4> |
| 1547 | <div class="paragraph"> |
| 1548 | <p>The reftable format assumes the vast majority of references are single |
| 1549 | object names valued with common prefixes, such as Gerrit Code Review’s |
| 1550 | <code>refs/changes/</code> namespace, GitHub’s <code>refs/pulls/</code> namespace, or many |
| 1551 | lightweight tags in the <code>refs/tags/</code> namespace.</p> |
| 1552 | </div> |
| 1553 | <div class="paragraph"> |
| 1554 | <p>Annotated tags storing the peeled object cost an additional object name per |
| 1555 | reference.</p> |
| 1556 | </div> |
| 1557 | </div> |
| 1558 | <div class="sect3"> |
| 1559 | <h4 id="_low_overhead">Low overhead</h4> |
| 1560 | <div class="paragraph"> |
| 1561 | <p>A reftable with very few references (e.g. git.git with 5 heads) is 269 |
| 1562 | bytes for reftable, vs. 332 bytes for packed-refs. This supports |
| 1563 | reftable scaling down for transaction logs (below).</p> |
| 1564 | </div> |
| 1565 | </div> |
| 1566 | <div class="sect3"> |
| 1567 | <h4 id="_block_size_2">Block size</h4> |
| 1568 | <div class="paragraph"> |
| 1569 | <p>For a Gerrit Code Review type repository with many change refs, larger |
| 1570 | block sizes (64 KiB) and less frequent restart points (every 64) yield |
| 1571 | better compression due to more references within the block compressing |
| 1572 | against the prior reference.</p> |
| 1573 | </div> |
| 1574 | <div class="paragraph"> |
| 1575 | <p>Larger block sizes reduce the index size, as the reftable will require |
| 1576 | fewer blocks to store the same number of references.</p> |
| 1577 | </div> |
| 1578 | </div> |
| 1579 | <div class="sect3"> |
| 1580 | <h4 id="_minimal_disk_seeks">Minimal disk seeks</h4> |
| 1581 | <div class="paragraph"> |
| 1582 | <p>Assuming the index block has been loaded into memory, binary searching |
| 1583 | for any single reference requires exactly 1 disk seek to load the |
| 1584 | containing block.</p> |
| 1585 | </div> |
| 1586 | </div> |
| 1587 | <div class="sect3"> |
| 1588 | <h4 id="_scans_and_lookups_dominate">Scans and lookups dominate</h4> |
| 1589 | <div class="paragraph"> |
| 1590 | <p>Scanning all references and lookup by name (or namespace such as |
| 1591 | <code>refs/heads/</code>) are the most common activities performed on repositories. |
| 1592 | Object names are stored directly with references to optimize this use case.</p> |
| 1593 | </div> |
| 1594 | </div> |
| 1595 | <div class="sect3"> |
| 1596 | <h4 id="_logs_are_infrequently_read">Logs are infrequently read</h4> |
| 1597 | <div class="paragraph"> |
| 1598 | <p>Logs are infrequently accessed, but can be large. Deflating log blocks |
| 1599 | saves disk space, with some increased penalty at read time.</p> |
| 1600 | </div> |
| 1601 | <div class="paragraph"> |
| 1602 | <p>Logs are stored in an isolated section from refs, reducing the burden on |
| 1603 | reference readers that want to ignore logs. Further, historical logs can |
| 1604 | be isolated into log-only files.</p> |
| 1605 | </div> |
| 1606 | </div> |
| 1607 | <div class="sect3"> |
| 1608 | <h4 id="_logs_are_read_backwards">Logs are read backwards</h4> |
| 1609 | <div class="paragraph"> |
| 1610 | <p>Logs are frequently accessed backwards (most recent N records for master |
| 1611 | to answer <code>master@{4}</code>), so log records are grouped by reference, and |
| 1612 | sorted descending by update index.</p> |
| 1613 | </div> |
| 1614 | </div> |
| 1615 | </div> |
| 1616 | <div class="sect2"> |
| 1617 | <h3 id="_repository_format">Repository format</h3> |
| 1618 | <div class="sect3"> |
| 1619 | <h4 id="_version_1">Version 1</h4> |
| 1620 | <div class="paragraph"> |
| 1621 | <p>A repository must set its <code>$GIT_DIR/config</code> to configure reftable:</p> |
| 1622 | </div> |
| 1623 | <div class="literalblock"> |
| 1624 | <div class="content"> |
| 1625 | <pre>[core] |
| 1626 | repositoryformatversion = 1 |
| 1627 | [extensions] |
| 1628 | refStorage = reftable</pre> |
| 1629 | </div> |
| 1630 | </div> |
| 1631 | </div> |
| 1632 | <div class="sect3"> |
| 1633 | <h4 id="_layout">Layout</h4> |
| 1634 | <div class="paragraph"> |
| 1635 | <p>A collection of reftable files are stored in the <code>$GIT_DIR/reftable/</code> directory. |
| 1636 | Their names should have a random element, such that each filename is globally |
| 1637 | unique; this helps avoid spurious failures on Windows, where open files cannot |
| 1638 | be removed or overwritten. It suggested to use |
| 1639 | <code>${min_update_index}-${max_update_index}-${random}.ref</code> as a naming convention.</p> |
| 1640 | </div> |
| 1641 | <div class="paragraph"> |
| 1642 | <p>Log-only files use the <code>.log</code> extension, while ref-only and mixed ref |
| 1643 | and log files use <code>.ref</code>. extension.</p> |
| 1644 | </div> |
| 1645 | <div class="paragraph"> |
| 1646 | <p>The stack ordering file is <code>$GIT_DIR/reftable/tables.list</code> and lists the |
| 1647 | current files, one per line, in order, from oldest (base) to newest |
| 1648 | (most recent):</p> |
| 1649 | </div> |
| 1650 | <div class="literalblock"> |
| 1651 | <div class="content"> |
| 1652 | <pre>$ cat .git/reftable/tables.list |
| 1653 | 00000001-00000001-RANDOM1.log |
| 1654 | 00000002-00000002-RANDOM2.ref |
| 1655 | 00000003-00000003-RANDOM3.ref</pre> |
| 1656 | </div> |
| 1657 | </div> |
| 1658 | <div class="paragraph"> |
| 1659 | <p>Readers must read <code>$GIT_DIR/reftable/tables.list</code> to determine which |
| 1660 | files are relevant right now, and search through the stack in reverse |
| 1661 | order (last reftable is examined first).</p> |
| 1662 | </div> |
| 1663 | <div class="paragraph"> |
| 1664 | <p>Reftable files not listed in <code>tables.list</code> may be new (and about to be |
| 1665 | added to the stack by the active writer), or ancient and ready to be |
| 1666 | pruned.</p> |
| 1667 | </div> |
| 1668 | </div> |
| 1669 | <div class="sect3"> |
| 1670 | <h4 id="_backward_compatibility">Backward compatibility</h4> |
| 1671 | <div class="paragraph"> |
| 1672 | <p>Older clients should continue to recognize the directory as a git |
| 1673 | repository so they don’t look for an enclosing repository in parent |
| 1674 | directories. To this end, a reftable-enabled repository must contain the |
| 1675 | following dummy files</p> |
| 1676 | </div> |
| 1677 | <div class="ulist"> |
| 1678 | <ul> |
| 1679 | <li> |
| 1680 | <p><code>.git/HEAD</code>, a regular file containing <code>ref: refs/heads/.invalid</code>.</p> |
| 1681 | </li> |
| 1682 | <li> |
| 1683 | <p><code>.git/refs/</code>, a directory</p> |
| 1684 | </li> |
| 1685 | <li> |
| 1686 | <p><code>.git/refs/heads</code>, a regular file</p> |
| 1687 | </li> |
| 1688 | </ul> |
| 1689 | </div> |
| 1690 | </div> |
| 1691 | <div class="sect3"> |
| 1692 | <h4 id="_readers">Readers</h4> |
| 1693 | <div class="paragraph"> |
| 1694 | <p>Readers can obtain a consistent snapshot of the reference space by |
| 1695 | following:</p> |
| 1696 | </div> |
| 1697 | <div class="olist arabic"> |
| 1698 | <ol class="arabic"> |
| 1699 | <li> |
| 1700 | <p>Open and read the <code>tables.list</code> file.</p> |
| 1701 | </li> |
| 1702 | <li> |
| 1703 | <p>Open each of the reftable files that it mentions.</p> |
| 1704 | </li> |
| 1705 | <li> |
| 1706 | <p>If any of the files is missing, goto 1.</p> |
| 1707 | </li> |
| 1708 | <li> |
| 1709 | <p>Read from the now-open files as long as necessary.</p> |
| 1710 | </li> |
| 1711 | </ol> |
| 1712 | </div> |
| 1713 | </div> |
| 1714 | <div class="sect3"> |
| 1715 | <h4 id="_update_transactions">Update transactions</h4> |
| 1716 | <div class="paragraph"> |
| 1717 | <p>Although reftables are immutable, mutations are supported by writing a |
| 1718 | new reftable and atomically appending it to the stack:</p> |
| 1719 | </div> |
| 1720 | <div class="olist arabic"> |
| 1721 | <ol class="arabic"> |
| 1722 | <li> |
| 1723 | <p>Acquire <code>tables.list.lock</code>.</p> |
| 1724 | </li> |
| 1725 | <li> |
| 1726 | <p>Read <code>tables.list</code> to determine current reftables.</p> |
| 1727 | </li> |
| 1728 | <li> |
| 1729 | <p>Select <code>update_index</code> to be most recent file’s |
| 1730 | <code>max_update_index + 1</code>.</p> |
| 1731 | </li> |
| 1732 | <li> |
| 1733 | <p>Prepare temp reftable <code>tmp_XXXXXX</code>, including log entries.</p> |
| 1734 | </li> |
| 1735 | <li> |
| 1736 | <p>Rename <code>tmp_XXXXXX</code> to <code>${update_index}-${update_index}-${random}.ref</code>.</p> |
| 1737 | </li> |
| 1738 | <li> |
| 1739 | <p>Copy <code>tables.list</code> to <code>tables.list.lock</code>, appending file from (5).</p> |
| 1740 | </li> |
| 1741 | <li> |
| 1742 | <p>Rename <code>tables.list.lock</code> to <code>tables.list</code>.</p> |
| 1743 | </li> |
| 1744 | </ol> |
| 1745 | </div> |
| 1746 | <div class="paragraph"> |
| 1747 | <p>During step 4 the new file’s <code>min_update_index</code> and <code>max_update_index</code> |
| 1748 | are both set to the <code>update_index</code> selected by step 3. All log records |
| 1749 | for the transaction use the same <code>update_index</code> in their keys. This |
| 1750 | enables later correlation of which references were updated by the same |
| 1751 | transaction.</p> |
| 1752 | </div> |
| 1753 | <div class="paragraph"> |
| 1754 | <p>Because a single <code>tables.list.lock</code> file is used to manage locking, the |
| 1755 | repository is single-threaded for writers. Writers may have to busy-spin |
| 1756 | (with backoff) around creating <code>tables.list.lock</code>, for up to an |
| 1757 | acceptable wait period, aborting if the repository is too busy to |
| 1758 | mutate. Application servers wrapped around repositories (e.g. Gerrit |
| 1759 | Code Review) can layer their own lock/wait queue to improve fairness to |
| 1760 | writers.</p> |
| 1761 | </div> |
| 1762 | </div> |
| 1763 | <div class="sect3"> |
| 1764 | <h4 id="_reference_deletions">Reference deletions</h4> |
| 1765 | <div class="paragraph"> |
| 1766 | <p>Deletion of any reference can be explicitly stored by setting the <code>type</code> |
| 1767 | to <code>0x0</code> and omitting the <code>value</code> field of the <code>ref_record</code>. This serves |
| 1768 | as a tombstone, overriding any assertions about the existence of the |
| 1769 | reference from earlier files in the stack.</p> |
| 1770 | </div> |
| 1771 | </div> |
| 1772 | <div class="sect3"> |
| 1773 | <h4 id="_compaction">Compaction</h4> |
| 1774 | <div class="paragraph"> |
| 1775 | <p>A partial stack of reftables can be compacted by merging references |
| 1776 | using a straightforward merge join across reftables, selecting the most |
| 1777 | recent value for output, and omitting deleted references that do not |
| 1778 | appear in remaining, lower reftables.</p> |
| 1779 | </div> |
| 1780 | <div class="paragraph"> |
| 1781 | <p>A compacted reftable should set its <code>min_update_index</code> to the smallest |
| 1782 | of the input files' <code>min_update_index</code>, and its <code>max_update_index</code> |
| 1783 | likewise to the largest input <code>max_update_index</code>.</p> |
| 1784 | </div> |
| 1785 | <div class="paragraph"> |
| 1786 | <p>For sake of illustration, assume the stack currently consists of |
| 1787 | reftable files (from oldest to newest): A, B, C, and D. The compactor is |
| 1788 | going to compact B and C, leaving A and D alone.</p> |
| 1789 | </div> |
| 1790 | <div class="olist arabic"> |
| 1791 | <ol class="arabic"> |
| 1792 | <li> |
| 1793 | <p>Obtain lock <code>tables.list.lock</code> and read the <code>tables.list</code> file.</p> |
| 1794 | </li> |
| 1795 | <li> |
| 1796 | <p>Obtain locks <code>B.lock</code> and <code>C.lock</code>. Ownership of these locks |
| 1797 | prevents other processes from trying to compact these files.</p> |
| 1798 | </li> |
| 1799 | <li> |
| 1800 | <p>Release <code>tables.list.lock</code>.</p> |
| 1801 | </li> |
| 1802 | <li> |
| 1803 | <p>Compact <code>B</code> and <code>C</code> into a temp file |
| 1804 | <code>${min_update_index}-${max_update_index}_XXXXXX</code>.</p> |
| 1805 | </li> |
| 1806 | <li> |
| 1807 | <p>Reacquire lock <code>tables.list.lock</code>.</p> |
| 1808 | </li> |
| 1809 | <li> |
| 1810 | <p>Verify that <code>B</code> and <code>C</code> are still in the stack, in that order. This |
| 1811 | should always be the case, assuming that other processes are adhering to |
| 1812 | the locking protocol.</p> |
| 1813 | </li> |
| 1814 | <li> |
| 1815 | <p>Rename <code>${min_update_index}-${max_update_index}_XXXXXX</code> to |
| 1816 | <code>${min_update_index}-${max_update_index}-${random}.ref</code>.</p> |
| 1817 | </li> |
| 1818 | <li> |
| 1819 | <p>Write the new stack to <code>tables.list.lock</code>, replacing <code>B</code> and <code>C</code> |
| 1820 | with the file from (4).</p> |
| 1821 | </li> |
| 1822 | <li> |
| 1823 | <p>Rename <code>tables.list.lock</code> to <code>tables.list</code>.</p> |
| 1824 | </li> |
| 1825 | <li> |
| 1826 | <p>Delete <code>B</code> and <code>C</code>, perhaps after a short sleep to avoid forcing |
| 1827 | readers to backtrack.</p> |
| 1828 | </li> |
| 1829 | </ol> |
| 1830 | </div> |
| 1831 | <div class="paragraph"> |
| 1832 | <p>This strategy permits compactions to proceed independently of updates.</p> |
| 1833 | </div> |
| 1834 | <div class="paragraph"> |
| 1835 | <p>Each reftable (compacted or not) is uniquely identified by its name, so |
| 1836 | open reftables can be cached by their name.</p> |
| 1837 | </div> |
| 1838 | </div> |
| 1839 | <div class="sect3"> |
| 1840 | <h4 id="_windows">Windows</h4> |
| 1841 | <div class="paragraph"> |
| 1842 | <p>On windows, and other systems that do not allow deleting or renaming to open |
| 1843 | files, compaction may succeed, but other readers may prevent obsolete tables |
| 1844 | from being deleted.</p> |
| 1845 | </div> |
| 1846 | <div class="paragraph"> |
| 1847 | <p>On these platforms, the following strategy can be followed: on closing a |
| 1848 | reftable stack, reload <code>tables.list</code>, and delete any tables no longer mentioned |
| 1849 | in <code>tables.list</code>.</p> |
| 1850 | </div> |
| 1851 | <div class="paragraph"> |
| 1852 | <p>Irregular program exit may still leave about unused files. In this case, a |
| 1853 | cleanup operation should proceed as follows:</p> |
| 1854 | </div> |
| 1855 | <div class="ulist"> |
| 1856 | <ul> |
| 1857 | <li> |
| 1858 | <p>take a lock <code>tables.list.lock</code> to prevent concurrent modifications</p> |
| 1859 | </li> |
| 1860 | <li> |
| 1861 | <p>refresh the reftable stack, by reading <code>tables.list</code></p> |
| 1862 | </li> |
| 1863 | <li> |
| 1864 | <p>for each <code>*.ref</code> file, remove it if</p> |
| 1865 | <div class="ulist"> |
| 1866 | <ul> |
| 1867 | <li> |
| 1868 | <p>it is not mentioned in <code>tables.list</code>, and</p> |
| 1869 | </li> |
| 1870 | <li> |
| 1871 | <p>its max update_index is not beyond the max update_index of the stack</p> |
| 1872 | </li> |
| 1873 | </ul> |
| 1874 | </div> |
| 1875 | </li> |
| 1876 | </ul> |
| 1877 | </div> |
| 1878 | </div> |
| 1879 | </div> |
| 1880 | <div class="sect2"> |
| 1881 | <h3 id="_alternatives_considered">Alternatives considered</h3> |
| 1882 | <div class="sect3"> |
| 1883 | <h4 id="_bzip_packed_refs">bzip packed-refs</h4> |
| 1884 | <div class="paragraph"> |
| 1885 | <p><code>bzip2</code> can significantly shrink a large packed-refs file (e.g. 62 MiB |
| 1886 | compresses to 23 MiB, 37%). However the bzip format does not support |
| 1887 | random access to a single reference. Readers must inflate and discard |
| 1888 | while performing a linear scan.</p> |
| 1889 | </div> |
| 1890 | <div class="paragraph"> |
| 1891 | <p>Breaking packed-refs into chunks (individually compressing each chunk) |
| 1892 | would reduce the amount of data a reader must inflate, but still leaves |
| 1893 | the problem of indexing chunks to support readers efficiently locating |
| 1894 | the correct chunk.</p> |
| 1895 | </div> |
| 1896 | <div class="paragraph"> |
| 1897 | <p>Given the compression achieved by reftable’s encoding, it does not seem |
| 1898 | necessary to add the complexity of bzip/gzip/zlib.</p> |
| 1899 | </div> |
| 1900 | </div> |
| 1901 | <div class="sect3"> |
| 1902 | <h4 id="_michael_haggertys_alternate_format">Michael Haggerty’s alternate format</h4> |
| 1903 | <div class="paragraph"> |
| 1904 | <p>Michael Haggerty proposed |
| 1905 | <a href="https://lore.kernel.org/git/CAMy9T_HCnyc1g8XWOOWhe7nN0aEFyyBskV2aOMb_fe%2BwGvEJ7A%40mail.gmail.com/">an |
| 1906 | alternate</a> format to reftable on the Git mailing list. This format uses |
| 1907 | smaller chunks, without the restart table, and avoids block alignment |
| 1908 | with padding. Reflog entries immediately follow each ref, and are thus |
| 1909 | interleaved between refs.</p> |
| 1910 | </div> |
| 1911 | <div class="paragraph"> |
| 1912 | <p>Performance testing indicates reftable is faster for lookups (51% |
| 1913 | faster, 11.2 usec vs. 5.4 usec), although reftable produces a slightly |
| 1914 | larger file (+ ~3.2%, 28.3M vs 29.2M):</p> |
| 1915 | </div> |
| 1916 | <table class="tableblock frame-all grid-all stretch"> |
| 1917 | <colgroup> |
| 1918 | <col style="width: 25%;"/> |
| 1919 | <col style="width: 25%;"/> |
| 1920 | <col style="width: 25%;"/> |
| 1921 | <col style="width: 25%;"/> |
| 1922 | </colgroup> |
| 1923 | <thead> |
| 1924 | <tr> |
| 1925 | <th class="tableblock halign-right valign-top">format</th> |
| 1926 | <th class="tableblock halign-right valign-top">size</th> |
| 1927 | <th class="tableblock halign-right valign-top">seek cold</th> |
| 1928 | <th class="tableblock halign-right valign-top">seek hot</th> |
| 1929 | </tr> |
| 1930 | </thead> |
| 1931 | <tbody> |
| 1932 | <tr> |
| 1933 | <td class="tableblock halign-right valign-top"><p class="tableblock">mh-alt</p></td> |
| 1934 | <td class="tableblock halign-right valign-top"><p class="tableblock">28.3 M</p></td> |
| 1935 | <td class="tableblock halign-right valign-top"><p class="tableblock">23.4 usec</p></td> |
| 1936 | <td class="tableblock halign-right valign-top"><p class="tableblock">11.2 usec</p></td> |
| 1937 | </tr> |
| 1938 | <tr> |
| 1939 | <td class="tableblock halign-right valign-top"><p class="tableblock">reftable</p></td> |
| 1940 | <td class="tableblock halign-right valign-top"><p class="tableblock">29.2 M</p></td> |
| 1941 | <td class="tableblock halign-right valign-top"><p class="tableblock">19.9 usec</p></td> |
| 1942 | <td class="tableblock halign-right valign-top"><p class="tableblock">5.4 usec</p></td> |
| 1943 | </tr> |
| 1944 | </tbody> |
| 1945 | </table> |
| 1946 | </div> |
| 1947 | <div class="sect3"> |
| 1948 | <h4 id="_jgit_ketch_reftree">JGit Ketch RefTree</h4> |
| 1949 | <div class="paragraph"> |
| 1950 | <p><a href="https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03073.html">JGit Ketch</a> |
| 1951 | proposed |
| 1952 | <a href="https://lore.kernel.org/git/CAJo%3DhJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg%40mail.gmail.com/">RefTree</a>, |
| 1953 | an encoding of references inside Git tree objects stored as part of the |
| 1954 | repository’s object database.</p> |
| 1955 | </div> |
| 1956 | <div class="paragraph"> |
| 1957 | <p>The RefTree format adds additional load on the object database storage |
| 1958 | layer (more loose objects, more objects in packs), and relies heavily on |
| 1959 | the packer’s delta compression to save space. Namespaces which are flat |
| 1960 | (e.g. thousands of tags in refs/tags) initially create very large loose |
| 1961 | objects, and so RefTree does not address the problem of copying many |
| 1962 | references to modify a handful.</p> |
| 1963 | </div> |
| 1964 | <div class="paragraph"> |
| 1965 | <p>Flat namespaces are not efficiently searchable in RefTree, as tree |
| 1966 | objects in canonical formatting cannot be binary searched. This fails |
| 1967 | the need to handle a large number of references in a single namespace, |
| 1968 | such as GitHub’s <code>refs/pulls</code>, or a project with many tags.</p> |
| 1969 | </div> |
| 1970 | </div> |
| 1971 | <div class="sect3"> |
| 1972 | <h4 id="_lmdb">LMDB</h4> |
| 1973 | <div class="paragraph"> |
| 1974 | <p>David Turner proposed |
| 1975 | <a href="https://lore.kernel.org/git/1455772670-21142-26-git-send-email-dturner@twopensource.com/">using |
| 1976 | LMDB</a>, as LMDB is lightweight (64k of runtime code) and GPL-compatible |
| 1977 | license.</p> |
| 1978 | </div> |
| 1979 | <div class="paragraph"> |
| 1980 | <p>A downside of LMDB is its reliance on a single C implementation. This |
| 1981 | makes embedding inside JGit (a popular reimplementation of Git) |
| 1982 | difficult, and hoisting onto virtual storage (for JGit DFS) virtually |
| 1983 | impossible.</p> |
| 1984 | </div> |
| 1985 | <div class="paragraph"> |
| 1986 | <p>A common format that can be supported by all major Git implementations |
| 1987 | (git-core, JGit, libgit2) is strongly preferred.</p> |
| 1988 | </div> |
| 1989 | </div> |
| 1990 | </div> |
| 1991 | </div> |
| 1992 | </div> |
| 1993 | </div> |
| 1994 | <div id="footer"> |
| 1995 | <div id="footer-text"> |
| 1996 | Last updated 2023-10-23 14:43:46 -0700 |
| 1997 | </div> |
| 1998 | </div> |
| 1999 | </body> |
| 2000 | </html> |