最近使用 AIX 7.1 从 Bitucket 下载代码的时候遇到了这个错误 fatal: write error: A file cannot be larger than the value set by ulimit.

$ git clone -b dev https://<username>:<password>@git.company.com/scm/vmcc/opensrc.git --depth 1
Cloning into 'opensrc'...
remote: Counting objects: 2390, done.
remote: Compressing objects: 100% (1546/1546), done.
fatal: write error: A file cannot be larger than the value set by ulimit.
fatal: index-pack failed

这是由于这个仓库里的文件太大，超过了 AIX 对于用户文件资源使用的上限。通过 ulimit -a 可以来查看。更多关于 ulimit 命令的使用 ulimit Command

$ ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         unlimited
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user)  unlimited

可以看到 file 有一个上限值 2097151。如果将它也改成 unlimited 应该就好了。

通过 root 用户可以访问到 limits 文件 /etc/security/limits(普通用户没权限访问)。

# 以下是这个文件里的部分内容

default:
        fsize = 2097151
        core = 2097151
        cpu = -1
        data = -1
        rss = 65536
        stack = 65536
        nofiles = 2000

将上述的值 fsize = 2097151 改成 fsize = -1 就将解除了文件块大小的限制了。修改完成后，重新登录，再次执行 ulimit -a

$ ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user)  unlimited

此时 file(blocks) 已经变成 unlimited 了。再次尝试 git clone

git clone -b dev https://<username>:<password>@git.company.com/scm/vmcc/opensrc.git --depth 1
Cloning into 'opensrc'...
remote: Counting objects: 2390, done.
remote: Compressing objects: 100% (1546/1546), done.
remote: Total 2390 (delta 763), reused 2369 (delta 763)
Receiving objects: 100% (2390/2390), 3.80 GiB | 3.92 MiB/s, done.
Resolving deltas: 100% (763/763), done.
Checking out files: 100% (3065/3065), done.

这次就成功了！

最近在我使用 Artifactory Enterprise 遇到了上传制品非常缓慢的问题，在经过与 IT，Artifactory 管理员一起合作终于解决这个问题，在此分享一下这个问题的解决过程。

如果你也遇到类似或许有所帮助。

问题描述

最近发现通过 Jenkins 往 Artifactory 里上传制品的时候偶尔出现上传非常缓慢的情况，尤其是当一个 Jenkins stage 里有多次上传，往往会在第二次上传的时候出现传输速度极为缓慢（KB/s ）。

问题排查和解决

我的构建环境和 Jenkins 都没有任何改动，所有的构建任务都出现了上传缓慢的情况，为了排除可能是使用 Artifactory plugin 的导致原因，通过 curl 命令来进行上传测试，也同样上传速度经常很慢。

那么问题就在 Artifactory 上面。

1. 是 Artifactory 最近升级了？
2. 还是 Artifactory 最近修改了什么设置？
3. 也许是 Artifactory 服务器的问题？

在跟 Artifactory 管理员进行了沟通之后，排除了以上 1，2 的可能。为了彻底排除是 Artifactory 的问题，通过 scp 进行拷贝的时候同样也出现了传输速度非常慢的情况，这问题就出现在网络上了。

这样需要 IT 来帮助排查网络问题了，最终 IT 建议更换网卡进行尝试（因为他们之前有遇到类似的情况），但这种情况会有短暂的网络中断，不过最终还是得到了管理者的同意。

幸运的是在更换网卡之后，Jenkins 往 Artifactory 传输制品的速度恢复了正常。

总结

处理次事件的一点点小小的总结：

由于这个问题涉及到几个团队，为了能够快速推进，此时明确说明问题，推测要有理有据，以及该问题导致了什么样的严重后果（比如影响发布）才能让相关人重视起来，否则大家都等着，没人回来解决问题。

当 Artifactory 管理推荐使用其他数据中心 instance，建议他们先尝试更换网卡；如果问题没有得到解决，在同一个数据中心创建另外一台服务器。如果问题还在，此时再考虑迁移到其他数据中心instance。这大大减少了作为用户去试错所带来的额外工作量。

I’m just documenting to myself that it was solved by following.

When I want to integrate the ESlint report with Jenkins. I encourage a problem

That is eslint-report.html display different with it on my local machine, and I also log to Jenkins server and grab the eslint-report.html to local, it works well.

I used HTML Publisher plugin to display the HTML report, but only the ESlint HTML report has problems other report work well, so I guess this problem may be caused by Jenkins.

Finally, I find it. (Stackoverflow URL)

Follow the below steps for solution

1. Open the Jenkin home page.
2. Go to Manage Jenkins.
3. Now go to Script Console.
4. And in that console paste the below statement and click on Run.

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "")

5. After that, it will load CSS and JS.

According to Jenkins’s new Content Security Policy and I saw No frames allowed.

That is exactly the error I get on chrome by right-clicking on Elements.

在使用 Git 提交代码之前，建议做以下这些设置。

叫指南有点夸张，因为它在有些情况下下不适用，比如你已经有了 .gitattributes 或 .editorconfig 等文件，那么有些设置就不用做了。

因此暂且叫他指北吧，它通常情况下还是很有用的。

废话不多说，看看都需要哪些设置吧。

1. 配置 name 和 email

# 注意，你需要将下面示例中我的 name 和 email 换成你自己的
$ git config --global user.name "shenxianpeng"
$ git config --global user.email "xianpeng.shen@gmail.com"

对于，我还推荐你设置头像，这样方便同事间的快速识别。

当你不设置头像的时候，只有把鼠标放到头像上才知道 Pull Request 的 Reviewers 是谁（来自于Bitubkcet）。

2. 设置 core.autocrlf=false

为了防止 CRLF(windows) 和 LF(UNIX/Linux/Mac) 的转换问题。为了避免在使用 Git 提交代码时出现历史被掩盖的问题，强烈建议每个使用 Git 的人执行以下命令

$ git config --global core.autocrlf false
# 检查并查看是否输出 "core.autocrlf=false"，这意味着命令设置成功。
$ git config --list

如果你的项目底下已经有了 .gitattributes 或 .editorconfig 文件，通常这些文件里面都有放置 CRLF 和 LF 的转换问题的设置项。

这时候你就不必特意执行命令 git config --global core.autocrlf false

3. 编写有规范的提交

我在之前的文章里分享过关于如何设置提交信息规范，请参看《Git提交信息和分支创建规范》。

4. 提交历史的压缩

比如你修改一个 bug，假设你通过 3 次提交到你的个人分支才把它改好。这时候你提 Pull Request 就会显示有三个提交。

如果提交历史不进行压缩，这个 PR 被合并到主分支后，以后别人看到你这个 bug 的修复就是去这三个 commits 里去一一查看，进行对比，才能知道到底修改了什么。

压缩提交历史就是将三次提交压缩成一次提交。

可以通过 git rebase 命令进行 commit 的压缩，比如将最近三次提交压缩成一次可以执行

git rebase -i HEAD~3

5. 删除已经 merge 的分支

有些 SCM，比如 Bitbucket 不支持默认勾选 Delete source branch after merging，这个问题终于在 Bitbucket 7.3 版本修复了。详见 BSERV-9254 和 BSERV-3272 （2013年创建的）。

注意在合并代码时勾选删除源分支这一选项，否则会造成大量的开发分支留在 Git 仓库下。

如果还需要哪些设置这里没有提到的，欢迎补充。

Why need branching naming convention

To better manage the branches on Git(I sued Bitbucket), integration with CI tool, Artifactory, and automation will be more simple and clear.

For example, good unified partition naming can help the team easily find and integrate without special processing. Therefore, you should unify the partition naming rules for all repositories.

Branches naming convention

main branch naming

In general, the main’s branch names most like master or main.

Development branch naming

I would name my development branch just called develop.

Bugfix and feature branches naming

For Bitbucket, it has default types of branches for use, like bugfix/, feature/.
So my bugfix, feature combine with the Jira key together, such as bugfix/ABC-1234 or feature/ABC-2345.

Hotfix and release branches naming

For hotfix and release, my naming convention always like release/1.1.0, hotfix/1.1.0.HF1.

Other branches

Maybe your Jira task ticket you don’t want to make it in bugfix or feature, you can name it to start with task, so the branch name is task/ABC-3456.

If you have to provide diagnostic build to custom, you can name your branch diag/ABC-5678.

Summary

Anyway, having a unified branch naming convention is very important for implement CI/CD and your whole team.

Related Reading: Git Branch Strategy (Chinese)

Problem

When you do CI with JFrog Artifactory when you want to download the entire folder artifacts, but maybe your IT doesn’t enable this function, whatever some seasons.

You can try the below JFrog Artifactory API to know if you’re using Artifactory whether allowed to download the entire folder artifacts.

just visit this API URL: https://den-artifactory.company.com/artifactory/api/archive/download/team-generic-release-den/project/abc/main/?archiveType=zip

You will see an error message returned if the Artifactory is not allowed to download the entire folder.

{
  "errors": [
    {
      "status": 403,
      "message": "Download Folder functionality is disabled."
    }
  ]
}

More details about the API could find here Retrieve Folder or Repository Archive

Workaround

So to be enabled to download entire folder artifacts, I found other JFrog Artifactory APIs provide a workaround.

How to download the entire folder artifacts programmatically? this post will show you how to use other Artifactory REST API to get a workaround.

1. Get All Artifacts Created in Date Range

API URL: Artifacts Created in Date Range

This is the snippet code I use this API

# download.sh

USERNAME=$1
PASSWORD=$2
REPO=$3

# which day ago do you want to download
N_DAY_AGO=$4
# today
START_TIME=$(($(date --date="$N_DAY_AGO days ago" +%s%N)/1000000))
END_TIME=$(($(date +%s%N)/1000000))

ARTIFACTORY=https://den-artifactory.company.com/artifactory

if [ ! -x "`which sha1sum`" ]; then echo "You need to have the 'sha1sum' command in your path."; exit 1; fi

RESULTS=`curl -s -X GET -u $USERNAME:$PASSWORD "$ARTIFACTORY/api/search/creation?from=$START_TIME&to=$END_TIME&repos=$REPO" | grep uri | awk '{print $3}' | sed s'/.$//' | sed s'/.$//' | sed -r 's/^.{1}//'`
echo $RESULTS

for RESULT in $RESULTS ; do
    echo "fetching path from $RESULT"
    PATH_TO_FILE=`curl -s -X GET -u $USERNAME:$PASSWORD $RESULT | grep downloadUri | awk '{print $3}' | sed s'/.$//' | sed s'/.$//' | sed -r 's/^.{1}//'`

	echo "download file path $PATH_TO_FILE"
  curl -u $USERNAME:$PASSWORD -O $PATH_TO_FILE
done

Then you just use this as: sh download.sh ${USERNAME} ${PASSWORD} ${REPO_PATH} ${N_DAY_AGO}

2. Get all artifacts matching the given Ant path pattern

More about this API see: Pattern Search

Take an example screenshot of pattern search:

Then you can use Shell, Python language to get the file path from the response, then use curl -u $USERNAME:$PASSWORD -O $PATH_TO_FILE command to download the file one by one.

If you have better solutions, suggestions, or questions, you can leave a comment.

What’s the problem?

Today I am having a problem where the Windows installer I created is not installing, and the following Windows installer box pops up.

But it works well in the previous build, and I didn’t make any code changes. It is strange, actually fix this problem is very easy but not easy to find.

How to fix it?

In my case, I just remove the space from my build folder naming. I have made follow mistakes:

My previous build name is v2.2.2.3500-da121sa-Developer, but for this build, I named it to v2.2.2.3500-32jkjdk - Developer

How to find the solution?

This problem takes me several hours until I google this article which inspired me.

Just like the above article, if I try to use the command line msiexec.exe other-commands ..., compare with the works installer will also quick to find the root cause.

I realized it immediately I should try to remove the spaces from the build folder… Wow, the installer back to work.

If this happens to you, I hope it also works for you, and leave a comment if it works for you.

本文适用的是 Gradle 来构建和适用 JaCoCo。

分别介绍了 build.gradle 的文件配置，执行测试和生成报告，报告参数说明，以及如何忽略指定的包或类从而影响测试覆盖率的结果。

build.gradle 文件配置

比如使用 gradle 来管理的项目可以在 build.gradle 里添加如下代码

plugins {
    id 'jacoco'
}

jacoco {
    toolVersion = "0.8.5"
}

test {
    useJUnitPlatform()
    exclude '**/**IgnoreTest.class'  // 如果有 test case 不通过，如有必要可以通过这样忽略掉
    finalizedBy jacocoTestReport       // report is always generated after tests run
}

jacocoTestReport {
    dependsOn test // tests are required to run before generating the report
    reports {
        xml.enabled true
        csv.enabled false
        html.destination file("${buildDir}/reports/jacoco")
    }
}

执行测试，生成代码覆盖率报告

然后执行 gradle test 就可以了。之后可以可以在 build\reports\jacoco 目录下找到报告了。

重点是如何分析报告。打开 index.html，报告显示如下：

报告参数说明

Read More

Python 不必多说，它是众多编程语言中最容易学习的动态类型语言。它的跨平台、易读、易写、丰富的 Packages 等众多特性，也是众多DevOps/测试/开发工程师是最常用的语言之一。

相信不少人用它完成了很多工作，但你是不是仅仅止步于功能的实现而忽略了去写出更加简洁，优美的 Pythonic 代码呢？

在我最开始用 Python 时，我还不知道 Pythonic 这个词，直到多年前一位资深的程序员在给我培训的时候提到了项目中有一些代码不够 Pythonic，需要重构。根据语境，我理解他的意思：就是 Python 的代码没有按照 Python 的方式来写。

什么是 Pythonic

充分利用 Python 语言的特性来产生清晰、简洁和可维护的代码。Pythonic 的意思是指代码不仅仅是语法正确，而是遵循 Python 社区的惯例，并以其预期的方式使用该语言。

举例

以下是 C/C++ 程序员的一段代码:

int a = 1;
int b = 100;
int total_sum = 0;
while (b >= a) {
    total_sum += a;
    a++;
}

如果没有学习 Python 编程模式，那么将上面的代码改用 Python 来写可能会是这样：

a = 1
b = 100
total_sum = 0
while b >= a:
    total_sum += a
    a += 1

如果用 Pythonic 的方式来写，应该是这样的：

total_sum = sum(range(1, 101))

再举个常见的例子，如果用 Java 可能是这样写出来一个 For 循环

for(int index=0; index < items.length; index++) {
    items[index].performAction();
}

在 Python中，使用以下方法会更干净一些：

for item in items:
    item.perform_action()

甚至是一个生成器表达式：

(item.some_attribute for item in items)

因此，从本质上讲，当有人说某件事情不符合 pythonic 时，他们是在说这段代码可以用一种更适合 Python 编码风格的方式来重新编写。
另外，去了解 Python Built-in Functions，而不是重新造轮子。

关于 Pythonic 的“官方介绍”

其实，Python 命令行里已经秘密“隐藏”了关于 Pythonic 的介绍。只要打开 Python 控制台，输入 import this，你就能看到：

C:\Users\xshen>python
Python 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>

直译过来是：Tim Peters 的《Python的禅意》

美丽的比丑陋的好。
明确的比含蓄的好。
简单的比复杂的好
复杂的比复杂的好
扁平的比嵌套的好。
稀疏比密集好。
可读性很重要。
特殊情况不特殊，不足以打破规则。
虽然实用性胜过纯粹性。
错误永远不应该默默地通过。
除非明确沉默。
在面对模棱两可的情况下，拒绝猜测的诱惑。
应该有一个--最好只有一个--明显的方法。
虽然这种方式一开始可能并不明显，除非你是荷兰人。
现在总比不做要好。
虽然从不比现在*好。
如果实现很难解释，那就是个坏主意。
如果实现很容易解释，它可能是个好主意。
命名空间是一个非常棒的想法--让我们做更多的命名空间!

关于 Pythonic 你 get 到了吗？

Problem

When you use Jenkins multibranch pipeline, you may want to have different default parameters settings for defferent branches build.

For example:

For develop/hotfix/release branches, except regular build, you also want to do some code analyzes, like code scanning, etc.
For other branches, like feature/bugfix or Pull Request that you just want to do a regular build.

So you need to have dynamic parameter settings for your multibranch pipeline job.

Solution

So for these cases, how to deal with Jenkins multibranch pipeline. Here are some code snippet that is works well in my Jenkinsfile.

def polarisValue = false
def blackduckValue = false

if (env.BRANCH_NAME.startsWith("develop") || env.BRANCH_NAME.startsWith("hotfix")
 || env.BRANCH_NAME.startsWith("release")) {
	polarisValue = true
	blackduckValue = true
}

 pipeline {
   agent { node { label 'gradle' } }

   parameters {
       booleanParam defaultValue: polarisValue, name: 'Polaris',  description: 'Uncheck to disable Polaris'
       booleanParam defaultValue: blackduckValue, name: 'BlackDuck', description: 'Uncheck to disable BD scan'
   }

   stages {
     // ...
   }
 }